The science behind the fiction - in conversation with Joe Letteri

Working with Steven Spielberg, Peter Jackson and James Cameron, Joe Letteri has built the astonishing photoreal worlds and iconic CG characters of Jurassic Park, Lord of the Rings, King Kong, the Planet of the Apes series and Avatar and is currently working on the highly anticipated Avatar sequels.

“Understanding how and why something happens, moves or works in the real-world gives you a deeper understanding of what make something look real in a virtual world,” he explains. “That has been the driving force behind the way I look at things. I want to see how far you can stretch the techniques and the technology in pursuit of realism.”

He grew up outside Pittsburgh, Pennsylvania, and excelled at science, maths, astronomy and physics. Attending the University of California, Berkeley, he gravitated toward media and film.

“I didn’t see a computer until I was at university,” he says. “But when I began using them to evaluate complex maths problems I became interested in graphics and how you could use computers to create organic shapes like clouds and mountains.”

Around that time mathematician Benoit Mandelbrot began using computer graphics to create and display fractal geometric images. “The infinite possibilities of visualising things with equations and making pictures out of data fascinated me.”

Letteri’s first job, at LA post house Metrolight, was designing logos and graphics for commercials and TV stations. It was enough to attract legendary VFX house ILM to hire him in 1990.

“This was my breakthrough. ILM were working on the opening shot for Star Trek VI: The Undiscovered Country and needed an explosion of the Klingon moon Praxis. They’d tried to do it as a practical effect but couldn’t get it to work.”

Letteri deployed his knowledge of fractals and the simulation of natural phenomena to design and animate an exploding ring of fire; “I was a huge Star Trek fan. I couldn’t believe my luck that this was the first shot I ever produced.”

Up next was Jurassic Park, the seminal picture that ushered in a new age of CGI. Spielberg had intended to use stop motion animation for the dinosaurs but changed his mind when shown a test of a CG T-Rex for which Letteri, working under ILM’s VFX supervisor Dennis Muren, helped establish a new level of realism.

‘The infinite possibilities of visualising things with equations and making pictures out of data fascinated me’

“The idea that you could actually write code [in Pixar’s 3D software RenderMan] to get the skin to look exactly how you wanted and how it reflects light was a revelation. But it wasn’t just about how real we could make [the dinosaurs] it was also about fitting them into the photography. We now had questions to answer about how these CG characters work dramatically within the context of the story.”

Computer graphics was emerging as a tool which combined photography and optics alongside with disciplines like biology for artists to construct increasingly more complex 3D creations. Letteri learnt cinematographic lighting techniques, drawing on his understanding of the physics of light transport, to make characters “believable for a story or setting that never been told or seen before.”

His credits at ILM include Casper, Mission: Impossible and the 1997 special edition of Star Wars.

Pixel by pixel
“Unlike special effects, where you built a model and moved it and photographed it, now you could manipulate the pixels directly,” he says. “It opened the door to a whole different way of thinking about the art, in that anything you could describe you could capture pixel by pixel.”

Jurassic Park’s success in making CG characters integral to the story had far reaching effects, ultimately inspiring director Peter Jackson to begin planning The Lord of the Rings series. In pre-production for the trilogy in 2001, Jackson hired Letteri to join his fledgling studio Weta Digital in Wellington and create Gollum.

“Gollum was the big attraction for me,” says Letteri. “He is one of literature’s great characters. What we needed to do was to make his skin look soft and translucent like human skin, something that hadn’t been done before.”

He devised a subsurface scattering technique to create the effect and also helped pioneer advanced performance capture with actor Andy Serkis, making Gollum the first character to appear alongside live-action characters in a way that didn’t differentiate between the two.

“Creating digital characters with that emotional range and subtlety has been the basis for all of the characters we’ve created since,” he says.

Performance capture evolution
The original contract with Weta Digital was for two years but Letteri never left. Now 61, he is its director, a shareholder, and responsible for all the work that goes that the studio generates including X-Men: The Last Stand, The Day the Earth Stood Still and I Robot.

Under his supervision, performance capture has evolved from a post-production process to a real-time one, allowing a director to shoot actors and digital characters and environments as if they are filming a live-action movie.

A highpoint for this was Avatar (2009), the landmark stereo film of the modern era. “Jim [Cameron] gave us the opportunity to take stock of everything we we’re doing and question all the assumptions of what worked and what didn’t,” Letteri recalls. “We built digital assets and environments, then combined these with the digital puppets being driven by the motion capture. It wasn’t just making believable CG characters it was the scale, detail, and complexity. We had to create an entire planetary ecosystem, with waterfalls and animals so that anywhere Jim wanted to point the camera he would see Pandora come to life in the viewfinder.”

Weta’s virtual production pipeline was refined over successive films. It was used extensively on The Adventures of Tintin: The Secret of the Unicorn and The BFG, both for Spielberg, and advanced again with The Planet of the Apes films which managed to take performance capture out of the studio into a forest and then even harsher conditions of snow and rain.

Under Letteri’s direction, Weta wrote in-house renderers, Manuka and Gazebo, which take their cue from physics to calculate how light interacts with each surface – down to the level of calculating wavelengths of light separately.

The facility’s artists and programmers have built on this to create a suite of tools that aim to exactly match CG renders with the actual lighting on set.

“On Avatar we used pretty unsophisticated lighting which was just good enough to allow Jim to sketch out his intent for composition,” Letteri explains. “What we really wanted was for it to be physically correct so that what the director sees on set translates in sync with the virtual lighting.”

PhysLight uses measurements of lumens and nits, colour temperature and ISOs taken from the camera. “We know the precise ND filters, the specific iris, shutter angle, sensor, framerate, even the serial number of the camera and lens. All this data is used to simulate and synchronise with the cinematography.”

‘I think we’ll reach a point where we won’t be able to tell what’s real and what’s not’

The Avatar sequels
All of these cutting-edge developments are being trained on Avatar 2, 3, 4 and 5 which are in production in New Zealand and LA for release beginning in 2020 until 2025. Everyone expects the sequels to break ground – and in multiple ways. Cameron has reportedly test shot sequences underwater. Yet with technology moving so fast, striking a balance between taking risks on innovation and using tried and tested routes is one the VFX super must weigh.

“The risk lies in not doing it,” declares Letteri. “We’ve never been conservative about what’s required of us to figure out a way of doing what hasn’t been done before. If you don’t try you are never going to get there.”

While not strictly a visual effects technique, high frame rates can be another tool to accentuate realism. Letteri worked on The Hobbit which was filmed at 48 frames a second although the hyper-vivid quality was not to everyone’s taste.

“We are testing [HFR] for Avatar,” Letteri confirms. “Audiences are used to seeing 24 frames per second where the intermediate frames go blank and their mind fills the space with an image. It engages the imagination in a certain way. There’s a case for high frame rates with stereo 3D and lots of fast motion and you want to erase motion blur. It depends on the intent of the filmmaker. They might experiment and see if it can be used it to the advantage of certain scenes within a film.”

He feels that audiences are more likely to suspend their disbelief seeing some physical effects like miniatures and stop motion “because there’s a physical reality to it even though they know it’s not real” but that CGI is unforgiving. “We have to get everything absolutely right and not slip up otherwise the fact that it’s not real will strike a more discordant note in the audience’s mind.”

That said, he says it’s often easier to solve a problem in a computer “because you quickly run into limitations when you do it physically.

“I think we’ll be able to reach a point where we won’t be able to tell what’s real and what’s not,” he says. “You will hit the limit of what you need every pixel to look like to do exactly what it would have done if you photographed that character. For some kinds of effects, I think we are already there.”

A keen astronomer who has long harboured fantasies about travelling to outer space, Avatar is a dream come true.

“When Jim said ‘let’s go to Alpha Centauri and make a whole new world” I couldn’t wait,” he says. “That’s where I’m going.”