From Gollum to Mowgli: How Andy Serkis pioneered performance capture

It’s extremely unusual, unprecedented even, for an actor - let alone an A-list star - to discuss the cutting edge of production techniques with so much passion and erudition. But Andy Serkis is not your usual filmmaker.

“Using game engine technology to achieve high quality rendering in realtime means that, once something is shot on set, there will be no post production,” he says. “I believe that is where we are headed and with VR, AR and interactive gaming platforms emerging it’s the most exciting time to be a storyteller.”

Serkis is synonymous with performance capture, the art of playing digitally enhanced characters which he has made his own over the last seventeen years.

Famously as Gollum in The Lord of the Rings and The Hobbit, Serkis has inhabited King Kong, also for director Peter Jackson, Captain Haddock in Steven Spielberg’s The Adventures of Tintin, Caesar in the Planet of the Apes trilogy and Supreme Leader Snoke in Star Wars: A New Hope and The Last Jedi.

In doing so he has done more than anyone to advance the technique from peripheral activity to fundamental to the way a film is shot, pioneering the transition from purely capturing motion to being truly capable of recording emotion.

Recently he has moved into directing, showing his versatility on well-received biographical drama Breathe before turning his attention to Mowgli, a big budget refresh of The Jungle Book for Warner Bros. with performance capture to the fore.

“I love Gollum, because he started all of this,” says Serkis.

“Gollum is the most incredible 21st century role for an actor. I will never see performance capture as a straitjacket… quite the opposite. I see actors or dancers being able to use performance capture as the most liberating tool. You are not just tied to a humanoid form. Now you can play anything.”

Behind the mask
It wasn’t always like this. In its early days there was “gross misunderstanding” about the technique, Serkis admits.

“It’s like wearing digital make-up or a digital costume and in every other respect it’s the same preparation and process as any actor would go through in creating any role. You are directed by the director, you are on set from day one to the end of the shoot, you are the complete author of that part. A director can have the animators take the data and augment away from the performance – but that is not real performance capture.”

He recalls the early reviews of Lord of the Rings questioning who was behind the mask. “Are they a dancer? A mime? A contortionist? and in every interview I found myself repeating that ‘No, I am playing a role and that is no different to any other actor.’”

The only distinction, if there is one, is that the performance captured actor has significant interaction with visual effects artists. “But not more so than actors wearing prosthetic make-up might have with the make-up department,” he says. “There’s no difference between actors playing simians in the original Planet of the Apes [1968] and what [VFX facility] Weta did with the reboot [Rise of the Planet of the Apes, 2011].”

The classic analogy is that of John Hurt, unrecognisable behind prosthetic make-up in The Elephant Man (1980) yet whose performance as John Merrick was critically lauded landing him a Bafta win and Golden Globe and Oscar nods for Best Actor.

No such recognition yet by the Academy for Serkis – or indeed any actor in a performance capture role – although each of the recent trilogy of Apes movies was nominated for Best VFX.

This year Serkis’ acclaimed performance in War for the Planet of the Apes went unheralded during Awards season, with Gary Oldman winning for Darkest Hour, his towering performance as Churchill smothered in visage-altering cosmetics.

One argument is that the Academy (and other awards bodies) find it problematic to distinguish between the actor’s performance and the digital artistry of VFX teams. Should the award go to Serkis alone when the work is augmented by animators?

“The fact of the matter is that visual effects are amply awarded,” says Serkis, who suggests a little discomfort with having the burden of an ambassadorial role for performance capture thrust upon him.

“It goes without saying that VFX artists are brilliant,” he says. “On every Marvel movie, for example, it’s clear that the work of the actors has been enhanced in almost every shot with a brilliant piece of CG. What I am talking about is the process of literally taking the performance of an actor and retargeting it onto a new physiognomy in a way which retains the fidelity of the performance, of the character. Of course, one could not exist without the other, but my point is that VFX artists have a VFX category.”

Pushing the boundaries
Which brings us to Mowgli, produced by Warner Bros and Serkis’ Imaginarium Productions and featuring Christian Bale, Benedict Cumberbatch, Naomi Harris, Cate Blanchett and Serkis, who plays Baloo.

It’s a production which promises to push the boundaries of performance capture possibilities further than ever before.

Serkis describes the advances made on the movie as a “true hybrid” in which different gradations of the performance capture process will inform the finished picture.

It’s a live action shot on jungle sets at Warner Bros’ Leavesden Studios and on location in South Africa. In that sense it’s more in the style of Planet of the Apes, than Disney’s 2016 The Jungle Book in which 3D animated characters inhabited a CG world.

“I have a real problem with talking animals on film,” Serkis explains of his reason for using performance capture for this project. “If you took a real photorealist tiger and put a voice on it I would find it emotionally unbelievable.”

The production for Mowgli has gone to great lengths to create realistic animals that will also emotionally connect with audiences. This stems from concept art for the creature design which is based on illustrations by Victorian biologist and botanist Robert Sterndale and on lithographs by Edward Detmold and his brother made for the first edition of The Jungle Book in 1903.

“If you imagine a timeline with, on one side, the actor’s face and on the other side you have a panther and the two sides morph to a point where you will see both panther and the human actor’s face. That for me was the fundamental way into the story.”

The cast spent three weeks having their facial performance captured in space set-up by Imaginarium at a hotel near Leavesden.

“I was very clear [to the cast] in rehearsal that you are creating a character not a photoreal animation,” says Serkis. “Some actors, including Benedict (who plays Shere Khan) and Tom Hollander (a hyena called Tabaqui) chose to bring a lot of physicality to the role. Others, like Peter Mullan (lion chief Akela), found that stillness was more important for their character.”

The physical movements of the cast were recorded by witness cameras and combined with the facial data at London facility Framestore where editor Mark Sanger (who had previously worked with Framestore on Gravity) made a first cut of the movie.

“From that cut we had the timing for scenes,” explains Serkis. The next stage involved capturing the physical performance of a troupe of actors from Imaginarium’s repertory on set at Leavesden. They wore polystyrene heads constructed to scale for each of the animal characters and were recorded alongside the live action performance of Rohan Chand as Mowgli.

“The data we captured here was useful for geography and choreography to block the scenes,” explains Serkis.

All 1,164 VFX shots were supervised and completed by Framestore which developed, with

Imaginarium, a new muscle system to allow animators to more directly replicate facial shapes and expressions.

It is the fine balance between the essence of an actor’s facial performance and the animator’s translation of that performance to create the final character that is most innovative.

“The industry is now on the verge of realtime facial capture,” says Serkis. “That’s the grail. Once you can do that, untethered by wires and cables to combine performance capture transparently with the rest of the physical performances in a virtual environment then we’re talking about being able to complete a production straight out of the box.”

Imaginarium is not alone in innovating such virtual production techniques. Everyone from VFX tools developer Foundry to Paramount Studios, Intel and James Cameron are looking to evolve technology that permits a director to film CG assets with live action live on set. What’s more, this streamlined method will in theory create one set of assets for purposing in all sorts of ways – ultimately a revenue generator for the studios involved.

“We are then in a world where the same assets can used across VR, AR or gaming,” says Serkis. “Assets can be used to create animated movies straight out of the box. Studios like Imaginarium will be best suited to take on the totality of virtual production from inception to final output.”

The art of performance
In his youth, Serkis showed talent as a painter, particularly in character drawing. He went to study visual art at Lancaster University “with no intention of becoming an actor” but gravitated toward theatre after designing posters for college drama shows then making props and gradually becoming more involved behind the scenes. By the end of his first year he had started acting, something that led, he says, to an epiphany.

“One of the great things about Lancaster Uni at that time was the degree of independence they gave students. Basically, you could create your own degree. So, I swapped visual arts for my own independent studies in theatre, set design and movement.”

Fascinated by using art and physicality to tell stories, Serkis started to put on shows using puppetry and directed and performed a one-man show in this final year based on Raymond Briggs’ Falklands War satire The Tin-Pot Foreign General and the Old Iron Woman.

“I then put that lot aside when I first started acting but after ten years I realised I had a huge desire not just to inhabit one character’s point of view but to tell a story from a visual and a storytelling and a directing perspective. I began to make short films and write scripts and then, by luck, I found myself in a world where suddenly all the skills I’d accumulated over the years fell into place.”

That world was Middle-earth, the fantasy setting for director Peter Jackson’s staggeringly ambitious adaptation of JRR Tolkein’s masterwork which many thought was unfilmable.

“I see actors or dancers being able to use performance capture as the most liberating tool. You are not just tied to a humanoid form. Now you can play anything”

Playing a creature like Gollum - which required him to watch his virtual performance in real time on a monitor - was “like being a puppeteer and a marionette at the same time.”

Art and painting have influenced all his creations, he says. The work of Francis Bacon, Egon Schiele and Leonardo da Vinci inspired his portrayal of Gollum.

“I found myself straddling acting, VFX and animation and I felt very comfortable in all those worlds and I felt comfortable connecting artistically with a very wide range of people.

“Throughout my acting career I’ve always connected to the physicality of a role and believed that to be just as important as a character’s emotional and psychological centres. So, when I started on Rings I found it relatively easy to launch wholeheartedly into a vocal and physical performance for Gollum.

“When I put on the motion capture suit and saw that character come to life it lit my fire in a big way. The first lifting of my arm in the suit and seeing my arm on screen as Gollum was extraordinary and exciting to me.

“Suddenly lots of things seemed to open up in terms of the potential of what an actor can do and of the stories that could be told from a filmmaking perspective.

“It can be used for digital resurrection (new performances from dead stars) and fake news too. All this technology can be used for good or bad.”

In tests for Mowgli, which went unused, Serkis recorded two people performing the front and back of an animal to create a quadruped. They retargeted the movement of a person’s hands to be the ears of a dog.

“There are so many ways of retargeting performance to drive character,” says Serkis. “The acting community is more and more excited about this and wanting to be part of it.”

Seeing the world in a different light
Serkis says he is motivated by projects that say something about the human condition “and helps others see the world in a different light.”

“In pure acting terms, I am paid to research and respond to a role and show it to an audience as truthfully as possible, but I’ve always believed that social change is a motive for creating art.”

He has long wanted to make a version of George Orwell’s Animal Farm, beginning preparation for it well before Mowgli. “It’s one of the most lengthy development projects there’s ever been,” he admits. “But we want to get it right. It’s a hugely important book and it’s even more pertinent now than when we first started working on it.”

There are parallels with filming Lord of the Rings which until Jackson came along had only been made once before as a 2D cartoon (1978). The only other attempt to film the political fable was a 1954 feature animation. Naturally, Serkis’ version will be performance captured.

“I struggled with the idea of doing a version of Jungle Book because of Orwell’s attitude towards Kipling. It opened up the whole debate about whether an artist can create an outstanding piece of literature that does connect with humanity - and yet have personal political views with which you might not necessarily agree. I had to wrangle with that before committing.”

In the event, Serkis’ PG13 certificate Mowgli, will have subtle political undertones.

“I don’t believe you can make Jungle Book today with being cognizant of modern politics and modern sensibilities,” he says. “Kipling was a very conflicted human – he was the most beloved author at the same time as being derided by Orwell (born the same year as The Jungle Book’s publication) as a jingoistic imperialist.

“We had to take that into account when making this film. Mowgli is an orphaned child who is stuck between two worlds. He can’t be accepted into the animal kingdom but neither can he accept all human customs. It’s about the how we treat the ‘otherness’ of fellow creatures and about man encroaching on the natural world and imposing his laws on the law of the jungle.”