Derek Bradley: Emotion capture and real-time facial performance capture in focus for Disney

“People can be surprised that a certain character in a film is digital – a prime example being the Winklevoss twins in The Social Network – but with a trained eye you can discern the nuances,” says Disney Research scientist Dr. Derek Bradley, who told IBC2016 all about it in a special conference session. “The biggest issue is when motion starts. You can make a pretty realistic render of a face, but this can break down if you want real subtlety, especially when rendering a known person.”

The Canadian studied computer science in Ottawa and gained a Ph.D. from the University of British Columbia, before being recruited by Disney Research in 2010.

“I had studied the 3D reconstruction of objects from cameras, in particular objects like clothing and faces that can deform over time,” he explains. “I was driven by a love of movies and VFX and hoped one day to apply these techniques to films. I was quite fortunate that when Disney was starting up a new office in Zurich, [Lab Director] Markus Gross happened to be in Vancouver, came by for a visit, saw my research and invited me to come over.”

It was the right call. Bradley’s efforts have helped result in the Medusa Performance Capture System, a mobile rig of cameras and lights coupled with proprietary software that can reconstruct actors’ faces in full motion, without using traditional motion-capture dots.

“We know where every point in an expression moves on the face, which is a key ingredient for VFX artists building a facial animation rig for an actor.” So key that Medusa has been used on such high-profile features as Star Wars: The Force Awakens, Maleficent, and The Jungle Book.

While these films are pushing towards the ultimate merger of production with post production – virtual production – Bradley thinks there’s a way to go yet.

Improvements ahead

“Real-time facial performance capture is not at the quality or resolution of offline processes. Every stage of the pipeline can still be improved. You can achieve a good representation of a face in real time, but it won’t exactly model their expressions.”

Among Bradley’s other projects is FaceDirector, a prototype intended to reduce the number of takes required for performance capture during principal photography.

“FaceDirector blends together different takes of an actor’s performance,” he explains. “You could have a single take where the actor is angry, another take where they are sad, then choose the moments of transition in post.”

Of Zurich, Bradley says there’s a growing tech scene with IBM, Google and several spin-offs located nearby. “One of the reasons Disney is based here was because it had the smart idea to partner with universities which were already strong in the areas it wanted to research. It chose to base next to the campus of the Swiss Federal Institute of Technology because it’s one of the world’s leading centres for computer graphics.”

“We know where every point in an expression moves on the face”

The ultimate goal is to create a fully realistic digital human, but to get there, filmmakers must overcome the ‘uncanny valley’, the curious disconnect between actual humans and digitally constructed ones.

“The trouble is no-one knows exactly what it is or how to fix it,” he says. “Most people believe it’s a problem around the eyes and mouth. We have focused some research into that area.”

If anyone can crack it, then you can bet it will be Bradley and the team of world-class scientists at Disney Research.

 This interview was first conducted for IBC2016.