Behind the Scenes: How sound and vision get under the skin in Nickel Boys

Staying true to the format of Colson Whitehead’s 2019 best-selling novel, Director RaMell Ross has shot Amazon MGM Studios feature Nickel Boys in a first-person point of view, presenting a unique challenge for sound design and dialogue mix.

Inspired by real events, the story follows two African American boys, Elwood and Turner, who are sent to an abusive reform school called the Nickel Academy in 1960s Florida. Because of the strict POV design, we rarely see the protagonists’ faces so the sound - both heightened and naturalistic - takes precedence.

Tony Volante, the lead Re-recording Mixer and co-supervising sound editor worked with Re-recording Mixer Dan Timmons to create the final Dolby Atmos mix. They previously teamed on Ross’ Academy Award-nominated documentary Hale County This Morning, This Evening (2018).

“What made this specific project special was RaMell’s choice to shoot Nickel Boys predominantly in first person perspective,” Volante says. “This would bring the soundscape to the forefront of the primary characters’ emotional storytelling.”

There were numerous discussions with Ross about how to generate concepts for presenting the first-person perspective artistically and authentically through sound. Throughout most of the film, it is quite obvious visually to the viewer who is in first person, but there are several exceptional moments when you think you’re not in first person until midway through a scene. That’s when the sound moves with the camera catching every detail and you realise you are the camera.”

By creating a specific dialogue sound in the mix for the camera’s voice through automatic dialogue replacement (ADR) breathing and unique dialogue treatment, the viewer is immersed in the world in front of and all around the camera. Moments like these are what keep pulling the viewer back to the first-person POV.

“Creating a sonic perspective that would accurately and entertainingly portray the first-person POV was a unique challenge for the dialogue mix,” explains Volante. “Initially, I began mixing the POV voice mono/centre while panning the other dialogue and the world around it.

“This sounded quite nice, but wasn’t really different from how I usually approach a traditional film mix. For Nickel Boys, we knew we wanted the first-person POV to have a unique sound relative to the rest of the dialogue in the film. The POV voice needed its own ‘space’ that separated it from the film slightly, but also subtle enough not to distract the viewer/listener out of the film.”

Their concept was to pull the voice slightly off the screen and hover it within the camera/viewer position. To begin the process, Volante started by creating a wider soundscape for the POV voice while monitoring in Atmos.

“This sounded good in Atmos, but down mixes did not capture the dialogue effect accurately,” he recalls. “It wasn’t going to be possible to do completely separate first-person dialogue treatments for all the different mix formats, so I wanted to absolutely make sure the stereo mix would accurately portray the POV.

“To come up with the POV sound, I made the decision to monitor in stereo with my Neumann NDH20 headphones. When mixing with headphones, I can more accurately hear the spatial differences - the proper amount of reverb and stereo imaging - that sometimes get masked within the mixing room’s acoustics.”

He experimented with various plug-in image settings to discover which ones sounded best. After listening to how these would upmix in 5.1 and Atmos, he used the one that also translated the best across all formats.

“Hearing how the effect translated from stereo to upmix rather than the more traditional approach of starting big and checking how it ends up downmixed in stereo, ultimately proved a more accurate approach.”

Dialogue Editor Michael Odmark created a set of tracks containing all the first-person POV dialogue. This allowed a customised spatial treatment during the mix for those particular clips.

“It was clear early on in the process that the POV clips needed to start as a stereo image before adding any additional treatment,” says Volante. “I added a slight stereo spread plug-in to the chain of these clips, followed by an upmix plug-in to spread the dialogue to multiple channels, including the surround channels.

“Some scenes needed a little extra spatial enhancement, so I added a reverb send for the POV that was used sparingly for an enhanced 3D ‘in your head’ effect. Despite having an immersive configuration that worked for most of the film, I discovered that minor tweaks were required throughout the mix depending on location or quality of the voice recording.”

Production Sound Mixer Mark LeBlanc recorded some production sound with a stereo/MS microphone during photography. There are many moments in the film where these recordings were used to enhance the spatial environments and capture a natural stereo spread of the background voices at Nickel Academy, further enhancing the POV perspective.

“I love panning, so this film was a dream to work on,” says Volante. “All dialogue in the film was panned to accurately portray the POV. Even the slightest off-centre image dialogue is panned, following the characters throughout the scene.”

Timmons used a similar technique for SFX and Foley mixing, panning not only hard EFX and Foley, but also unconventionally shifting the background pan viewpoint to enhance the POV camera motions.

Volante says his colleague’s sound design during the ‘White House’ punishment scene is a highlight in the way of immersing the viewer into Ellwood’s POV. The viewer doesn’t see his beating but sound conveys the brutal whipping while black and white archival footage is shown.

Source music also played an important role in portraying the first-person narrative. Special care in panning was taken for the music tracks playing out of a radio or from a record player. Panning was, needless to say, very active during this mix.

The dynamic score, by Scott Alario and Alex Somers, was delivered in stereo stems. Volante says: “For music, especially for a movie like this, I prefer stereo stems to 5.1 stems since I can be more precise with the panning and channel placement. I like catching movement on screen and panning components in time with the score. Using the stereo stems, I was able to create spatial movement inside scenes, which further enhanced the sensory experience of the first-person POV.”

The feeling of sight

Cinematographer Jomo Fray played an intimate role in adapting the story. “From the first time I read the script I saw every single moment in the movie in a first-person perspective,” he says. “We constantly asked ourselves how could we manipulate traditional film language to work from a sentient perspective.”

By this, he means images that don’t just convey what is being seen from the point of view of the protagonists but how they are feeling too.

“I feel like the promise of cinema is the ability to walk in another person’s shoes. To feel what it’s like to be another human on Earth. What’s so unique about this film is that it truly does invite you into being in the body of a young black boy during the ‘Jim Crow’ years.”

He is referring to laws introduced in the Southern States that enforced racial segregation, ‘Jim Crow’ being a pejorative term for an African American.

“I want this film to invite you into living concurrently with them and their thoughts and their feelings as they move through the world.”

The technique they evolved was to shoot long takes to maintain a flow of camera with actor and to capture details to transition in and out of scenes. These details or inserts as Fray calls them include a deck of cards being shuffled and a gold bracelet.

“Inserts are the visual idea of the things that our mind remembers after an event that are more fragmentary, but still are a part of that memory,” Fray explains.

“These details became really important to us, not only for the edit, but also to try to describe the experience of sight,” he says, “the way that sometimes we hyper-fixate on things especially in moments of inhumanity.”

“After a car crash people rarely remember the impact itself as if our brain deletes the memory. There’s some aspect of that that feels tied to trauma. We tried to build in that kind of camera language throughout the film as Elwood tries to interpret what has been a very traumatic experience for him.”

Fray shot on a Sony Venice with Panavision large-format spherical primes often in shallow focus and with a 4:3 aspect ratio throughout. The format helped integrate archival collages of American history including the moon shot program of the late 1960s and black and white stills of Black children who may have endured the same treatment over the years.

“For me, the use of archive felt like a cascading of thoughts so using 4:3 was a way of trying to lessen the artifice and immerse the audience into that,” he says.

Combining a 4:3 aspect ratio with shallow focus was also a means of articulating the feeling of sight. “Humans have an incredibly wide field of view but our experience of looking is of our brain forming selective focus on where we look. Our brain will perceive something to be in focus whereas everything else is out of focus.

“RaMell and I wanted to capture that feeling of selective focus and of the way that the brain then puts together meaning in what we see. Hopefully, we are inviting the audience to not just see through these boy’s eyes but be in their thought process.”

Of the lens choice he says: “The way that they shot volume felt really special. Especially for a movie like this, where you’re seeing from the eyes of a character, it’s really important to have a sense of presence about the relationship between the camera’s eye and the rest of the space around you.”

Fray recalls a moment in production when Aunjanue Ellis-Taylor (playing Elwood’s grandmother) has to give Elwood some devastating news. “As we were shooting that scene, I’m thinking as Elwood and I found it really hard to look her in the eye. When I’m hearing her start to say something that I know is painful for her to have to say, my camera’s gaze drifts away.

“There was a moment of silence and Aunjanue did something unscripted. She put her hand out and she said, ‘Elwood, look at me son’. As an operator, I had to look back. I had to meet her gaze. After that take that, we all understood with clarity why a sentient perspective is so interesting.

“It isn’t just that the image is inside the scene, it’s that the image itself has to respond and react to the vulnerability that the actors are giving. There’s just such a deep intimacy that is created here not only for the viewer, but even for us as operators. As an operator you compose in a different way someone is acting as a mother and physically hugging you as a mother. That changes how you think of composing the scene.”

In one scene set at a bar, an adult Elwood meets a former inmate from Nickel. Daveed Diggs, the actor playing older Elwood suggested to Fray that he open up the right side of his shoulder to allow the other character to come into shot. He then suggested that he close his shoulder when Elwood starts feeling isolated by the conversation. Doing so pulled the secondary character out of the frame.

“It was co-authoring back and forth,” says Fray. “As the cinematographer, I was invited to connect and feel like a partner to the actors. There are also shots where the actors were invited to have a sense of co-authorship of the image. How they moved their body fundamentally changed the image.”

This is particularly the case for scenes set around 2010 which depict the older Elwood. To convey this the camera is mounted on the actor themselves using a rig called a SnorriCam. It was positioned to keep the back of their head in shot with the effect of giving the character, decades on and still traumatised, a feeling of disconnect with his younger self.

“The effect is what I call a second person perspective which is being able to see yourself in space but with a slight dissociation with yourself that just felt right for how trauma is remembered.”