Reframing the future

There are exciting development at Sigmedia in Trinity College Dublin and one of its spinout companies has just been bought by Google

IMPROVING OUR understanding of the world at an atomic scale, helping to provide Oscar-winning special effects for such movies as The Matrixand X-Men, counting lobsters on the sea-bed off Galway and producing 3D training videos for the surgeons of tomorrow are just a few of the activities being undertaken by the Signal Processing and Media Applications Group (Sigmedia) at Trinity College Dublin.

The group was established in 1998 when founder Dr Anil Kokaram joined the electrical and electronic engineering department at the university as a lecturer. It has since received two significant funding awards from Science Foundation Ireland.

Sigmedia’s research centres on digital signal processing technology and three main areas of work: digital cinema, multimedia information retrieval and video over wireless.

It has been involved with several projects funded by the European Union and has helped to create new cinema post-production tools. Its motion estimation technology has been used around the world on the special effects in such films as The Lord of the Rings, Spider-Man,and Harry Potter.

Signal processing is a fundamental component of this work. The group exploits knowledge from statistics, applied mathematics, computer vision, and image-, video- and audio-processing in order to solve unique problems in several domains.

Sigmedia comprises between 10 and 15 people, depending on the time of year. During the summer, the complement is augmented by undergraduate interns from Trinity as well as from universities in France, Germany, Trinidad and Tobago. Currently there are three post-doctoral and seven PhD students with the group.

Before he came to Trinity, Kokaram studied at Cambridge University, and his main research area was motion picture restoration.

“My work was in making old movies look like new. With the advent of DVD technology, there was a need to improve the quality of old movies,” he says. “Also, Sky TV was going digital around that time and had nothing to show. It became very important to have an automated process to digitise video instead of the very slow manual process that existed until then.”

The group moved into the restoration of sports archive material too and then into post-production. That led to a tie-up with leading visual effects company, the Foundry. "They recognised that we could help with special effects in terms of filling in missing frames and so on. We worked on the last two Matrixfilms with them and we got an Oscar in 2007 for our work with them," he says.

The fundamental technology underpinning the work of Sigmedia is signal processing and this has applications across a wide range of areas.

“There is an increasing use of video in the life sciences area,” Kokaram explains. “For example, the atomic force microscope has been around for quite some time now. It shows things at the nano scale, enables us to see things at an atomic level, which we couldn’t before. It can look at the physical properties of substances, such as friction between fluids like oil and water. But if you look at the pictures it produces, they are very noisy and grainy. We improve those pictures.”

The group is also involved in a particularly interesting project in conjunction with the university’s zoology department and the Marine Institute in Galway.

“The institute carries out regular sea-bed surveys where they take a camera, put it on a sledge and drag it along the sea-bed,” he explains. “At present, someone has to sit down and watch hours of this video and try to interpret what they are seeing. They have to guess how much of the sea-bed is sand, how much is gravel, what has been damaged by trawler nets and so on. They also have to count the number of lobsters – and this is not easy.”

This is where the advanced signal processing technology developed by the group comes in. “When you look at the video that comes back from the sea-bed, it’s all a bit greenish with a few black blobs here and there,” says Kokaram.

“And then you see a pinkish bit that might be a lobster. Lobsters tend to stay near their holes, so when you see a circular dark area with a pink blob nearby, that’s probably a lobster.

“What we have done is figure out a set of numbers from the video signals which are lobsters. What we do is turn the pictures into a probability surface and anything over 90 per cent is a lobster. At that point, we are ready to count.”

One piece of Science Foundation Ireland work being undertaken by the group takes it into the computer-generated image (CGI) area.

“A key part of the SFI project involves putting the user back into the CGI loop,” says Kokaram. “For example, at present if a user wants to put a character from one scene into a different scene, they draw a line around it and put it there.

“It would be good if the computer could help the user with that cut-out and a situation where the machine can learn from the user. That’s one area that we are working on.”

He points out that the group doesn’t work on the graphics themselves but on the background production process. “We prepare images so that they look good when other images are put on them,” he explains.

“For example, a filmmaker might synthesise some dinosaurs and wants to put them in a field. If the field has been filmed during the day, the image will have different grain in it than the dinosaurs and the image won’t look right. We take the grain out and then put it back in over the finished image so it looks right.”

Another SFI-funded project involves the group in 3D film-making. “We have two people looking at media post-production and one of them is looking at stereoscopic movies. We did the first 3D movie shoot in Ireland as part of this. We are now working with the College of Surgeons on creating a stereoscopic training video on surface anatomy for students.”

Another strand of this research relates to correcting errors in the 3D video process to ensure that the cameras are in tune with each other at all times.

While the commercial applications of the Sigmedia group’s research are clear, its true potential was illustrated in March when Google bought Green Parrot Pictures, a spinout company from the group, in a bid to improve the quality of video uploaded to YouTube.

Green Parrot specialises in image processing to improve sharpness and camera shake, among other things, and Google intends to use it to enhance the look of videos posted on YouTube while using bandwidth more efficiently.

Jeremy Doig, Google’s director of video technology, said at the time of the takeover: “What if there was a technology that could improve the quality of such videos – sharpening the image, reducing visual noise and rendering a higher-quality, steadier video – all while your video is simply being uploaded to the site? You can imagine how excited we were when we discovered a small, ambitious company based in Ireland that can do exactly this.”

Kokaram, chief executive of Green Parrot, says: “Google took an interest in the company some time ago because our technology could improve the quality of the images on YouTube. We’re very excited to join Google. I will continue working with Green Parrot and I will remain here in TCD pushing our research into the life sciences and other areas and help create more successful spinout companies.”