IBC2021 Accelerator: Immersive Audio and Sound Imagery

It is very tempting to think of immersive audio as a done deal within the industry. Great strides have been made in audio reproduction in recent years, with consumer deployments of 5.1 and Dolby Atmos becoming increasingly mass market and no longer simply the preserve of the AV enthusiast.

However, as this fascinating Accelerator focusing on Immersive Audio and Sound Imagery proves, there is a lot that can still be achieved. New object-oriented audio technologies not only raise the bar for high-end audio reproduction, but also hold out the potential to vastly improve a range of areas of more pragmatic interest to broadcasters.

“We see the delivery of object audio not in order to fly it around the living room, but so viewers can make their own mixes and be able to choose from a list of languages,” comments Karl Petermichl, Technische Direktion at Austrian broadcaster ORF, one of the Co-Champions on the project.

The mix of pragmatic solution and high-end innovation is, of course, one of the factors that makes the IBC Accelerators so compelling and draws in Champions and Participants from such a wide variety of sectors as a result. This one in particular has a broad range of Champions from industry and academia working together to produce three binaural POCs for the Show that will showcase some of the new possibilities engendered by immersive audio from live music performance through 360-video and on to spoken word dramatisations.

“We see the delivery of object audio not in order to fly it around the living room, but so viewers can make their own mixes and be able to choose from a list of languages,” Karl Petermichl, ORF.

Initially pitched at the IBC Accelerators Kickstart event by independent media consultant and project lead Benjamin Schwartz, his DIY soundscape experiment ‘Being There’ helped inspire the wider scope of this Challenge. “I was stranded at my mother’s house in March 2020 and deeply disappointed to miss a concert due to lockdown,” he explains, which is when he realised he could leverage his experience working with the Ultra HD Forum and the 8K Association.

“I was looking for a solution to address millions stranded like me in a house with just a regular HD TV and stereo sound. I wanted a solution that would make a difference for everyone. Knowing that the sound mix in music videos is permanently fixed during a performance, even if there are varied camera angles, I started working on the ‘Being There’ project: breaking that un-said rule of a static audio mix.”

To further explore where cutting-edge audio innovation can be applied to this basic concept, one area of the POC will focus on a fully immersive music experience, showcasing six degrees of freedom, room spatialisation and 360VR elements. The idea is that dynamic soundscapes will accompany the camera and, thanks to input from project Participants Omnilive (video) and MagicBeans (audio) , the viewer will be able to select the camera angle and even move around within the 360-degree space. As they do thisthe audio perspective will match their movement thanks in part to the pre-release 6DOF volumetric audio feature of MagicBeans’ forthcoming RoundHead software.

After hearing an ideal piece of tango-based music that could demonstrate this at a concert, Schwartz approached the composer and cellist, Soizic Chevrant Breton, to re-orchestrate it for Cello, Violin, Flute and Bass Clarinet, which she obligingly did. “We put together a group of professional musicians and recorded it,” he says, “so I suppose you could call this a world première of ‘Comme un Tango’.”

The key software brought into the Accelerator with the express aim of achieving the spatial mix is Braud.io, a spin-out from Co-Champion Kings College London, which has developed a subset of technologies that it hopes will be able to underpin the next generation of audio. In addition to its application within the music R&D of this Accelerator, it will also be implemented into the POC’s spoken word use case. This will focus on a multi-narrative audio play, specially commissioned by fellow Champion Audible and recorded at Twickenham Film Studios, another Co-Champion of this Challenge. Braudi.io’s co-founder is Zoran Cvetkovic, Professor of Signal Processing at KCL, and he lists two main challenges in the field of broadcast immersive audio.

Read more IBC Accelerators: 5G LBXR

The first, how to provide a stable auditory perspective for a listener moving around the listening environment, is beyond the scope of this Accelerator; hence the decision to present the POCs binaurally at IBC. The second is being very much tackled though and relates to techniques for capturing the room acoustics.

“When you observe an acoustic event in enclosed spaces, what you hear is the initial wave and then the first order reflections and second order reflections and so on of the walls and objects,” explains Cvetkovic. “It is via the processing of the relationships between these different reflections that the human auditory system creates a perception of where the objects in the space are. The more of these reflections we render the better it is, but that is numerically prohibitive to be done with proper accuracy in real time and is made more difficult by moving objects. However, we have developed technology that allows for synthesising most of it in real time using modest computational resources.”

Broadcast use cases

The team behind Braud.io originally developed the technology with classical music in mind before realising its potential in immersive environments. And it’s the prospect of taking that potential, and object-oriented audio in general, and applying it to the broadcast environment that has ORF’s Petermichl excited.

“The way it automatically adapts to the actual listening environment is very important for us,” he says. “In the era of smartphones and big TVs, you never know where your where your programme is listened to, on the underground with earphones in, or on a 100-inch Smart TV in a big living room in a cinema-like experience. Synthesising and adapting a given configuration to the actual point of listening is a really important task for us.”

Petermichl has been involved in two recent EBU recommendations, R153 and R154, that have looked carefully at the subject of object-oriented audio in the context of broadcast production audio templates for UHD/HDR file exchange. Building on that work, he sees object-oriented audio as a more complex offering which will actually translate into a more simple and effective experience in the living room.

It will also hopefully address four key issues for broadcasters. The perennial problem of variation in loudness and viewers not being able to hear the centre channel; improving language versatility; providing interactivity for narrative purposes, and being able to deliver stems to provide a more immersive mix.

The importance of Being There

Immersive Audio and Sound Imagery is an active Accelerator that is constantly retargeting itself. While originally looking to provide dynamic soundscapes that will accompany camera movement, as Schwartz relates the list of innovations has grown as the number of active participants has. Trialing a new approach for sound source re-specialisation as well as a six DoF approach to sound are now both on the cards. At its core though, for all the multiple benefits that may cascade out of it from solving accessibility issues to new forms of storytelling, it remains fundamentally an experiential POC.

“We see our work as a stepping-stone to immersive technologies, which might require specific equipment at the customer end,” he concludes. “We don’t know if our current iteration of technology will just play a proof-of-concept role to advance the immersive agenda or be used for several years, maybe even commercially. That wasn’t our initial purpose. We just want to rediscover that wonderful sense of Being There when the audience understands the artistic intent of a live performer.”