Bringing more realism to animated films

A few miles up the road and to the left from our Siemens EDA base in Wilsonville, Oregon is a “hidden” gem – LAIKA stop-action animation studio. Within those walls are the talented group that brought us unforgettable and award-winning features such as Coraline, ParaNorman, The Boxtrolls, Kubo and the Two Strings and Missing Link. The folks at the studio use a classic 3D animation technique called stop-motion which is pretty rare in these days of film making. They can spend all week capturing 5 seconds of film at 24 frames per second. My favorite stop-action film from childhood is Rudolf the Red-Nosed Reindeer. Sure, you can tell it is an animation, but I still watch it every year because I easily suspend disbelief and fall into the story. That’s what a good stop-action film does, and it holds up over decades.

The LAIKA team is working on their new film Wildwood, which is a fairy tale that is set in a forest outside Portland Oregon. It is based on a series of children’s novels by Colin Meloy, who is a member of the group The Decemberists, also based in Portland, Oregon. What possible use does this stop-action team for artificial intelligence? It would seem that this age-old technique is just a mix of capturing tedious small movements combined with a lot of patience. But, LAIKA teamed up with Intel (in the same town as LAIKA) to develop tools that are driven by machine learning and AI. These tools learn how to predict shapes in relation to surrounding images during the digital rotoscoping process. This speeds up the process and uses less data. The Intel team set out to apply their AI knowledge to the specific problems that LAIKA was trying to address.

I wanted to learn more and luckily, a colleague found a video created by Intel that includes a discussion with two members of the LAIKA team (thanks Nick!). Here is what I learned:

• They use puppets within a staged scene that have 3D-printed faces to capture different facial performances for each shot. They use tracing (rotoscoping) to draw lines down the middle of the faces to create segments so that they can change something or leave something alone in the area in post-production. Like taping your house before painting. This technique makes the puppet’s expressions more realistic.
• They wanted to automate the rotoscoping task, which is tedious and no animator wants to spend time on this non-creative activity.
• Intel created a process that employs machine learning to perform image segmentation and the team was looking to apply this technique to a real-world process. So, they teamed together with LAIKA to solve the rotoscoping issue. Intel learned how the artists do their work which was key to the project.
• For training data, they only needed 5 or 6 shots of the face to learn how to detect the lines, which is different from the normal idea that more data is better.
• Machine learning examines the segments on each side of the lines, finds an issue, takes a pass at the correction. But the artist has the last word on the change. The artist can use tools to automate correcting the frame. This solution helps accelerate the tasks, but the artists are always in control.
• Rotoscoping is also used to isolate the characters from the background allowing the team to add digital content. For example, instead of creating thousands of puppets for a crowd scene, they can add in digital versions against a green screen area combined with the main character puppets. That means automatic full puppet rotoscoping, which isolates them, is necessary. This technology does not exist yet, but the teams think machine learning might be applied to this problem.

The bottom line for LAIKA and Intel is they want to automate repetitive tasks with AI technology, but the artists are always the key to the film making process and they can never be replaced by AI. And, I am amazed at all the players in this project are from Portland, Oregon area; the story, band, Intel, and LAIKA. That’s synergy at its best.

When you watch Wildwood, instead of trying to figure out what star is voicing a character, think about how AI brought you a new gem to watch over and over again. I am hoping that the LAIKA team gets the awards they will certainly get nominated for and who knows, Intel might pick up some technical awards as well.

Look for the video “LAIKA and Intel Use Machine Learning to Make a Dramatic Impact to Movie Making,” to learn more about this project.