Immersive audio: consumer tech catches up with production potential

Six years on from the first-ever live Dolby Atmos broadcast, which took place at the Rio Olympic Games, it would be fair to state that immersive audio has now become mainstream. As an element of deliverable expectations for streaming services like Netflix, as well as a regular choice for live sports and music productions, immersive audio’s benefits are well-established across the full spectrum of content.

Henry Goodman, director of product management at Calrec Audio – which provided the mixing console for that historic broadcast in 2016 – pinpoints the transition that has occurred: “The difference now is that consumer technology has caught up. Immersive and binaural content can be streamed to headphones via mobile devices, and audio soundbars can replicate an immersive mix without requiring an additional seven speakers. This is driving consumer expectation for more immersive and more personalised content, especially in live sports, and manufacturers like Calrec have been building features into their equipment to help cater for this increased demand.”

• Watch the Accelerator video Immersive audio and sound imagery

Part of the challenge for manufacturers trying to keep pace with demand is that mixing configurations and techniques is still the subject of considerable variation. While some broadcasters are embracing immersive audio by creating a 5.1 mix and adding an extra height mix for atmosphere, others are exploring more elaborate configurations that involve more complex monitoring and audio object management. One especially welcome consequence of all this is that there is a tremendous amount of creativity taking place – and none more so that in the area of sports broadcasting.

Pushing the immersive audio envelope

The decision by Olympic Broadcasting Services (OBS) to capture the recent Winter Games with a 5.1.4 audio set-up has undoubtedly helped to raise the profile of immersive audio with the public. But from a production perspective, it is a further continuation of what broadcasters have long known – that sport is a perfect showcase for immersive sound, as it fully complements and augments shared human experiences.

“A big part of any live experience is what it sounds like because it’s part of the collective experience,” says Goodman. “This is the case at a concert, show or play, but it is especially true at a large sporting event. The swell of the crowd at a football match is a gauge of how exciting it is and it’s part of the shared experience. Sports broadcasters mixing in immersive tend to put the viewer in the midst of the crowd to share in that experience. However, more broadcasters are pushing the envelope further with live object-based mixing and personalisation, and assistive technologies are giving them the tools to do that.”

While UHD picture quality transports crucial information more clearly, audio’s role is to convey emotions.

Of course, more complex applications involving Next Generation Audio (NGA) – the increasingly favoured catch-all term for various immersive audio technologies – bring with them additional requirements in terms of digital signal processing (DSP) to cater to the increased number of channels, and more control to mix and monitor different outputs. Hence Calrec’s own latest development in this area – the ImPulse IP core. This adds 3D immersive path widths of up to 7.1.4, as well as planning for next-generation audio applications with comprehensive downmixing built-in. It also provides 1458 DSP processing paths, which are freely available on up to four mixing environments from the same core.

Looking ahead, Goodman observes that: “As NGA coverage continues to develop and object-based mixes become more commonplace, manufacturers like Calrec need to be more aware of how metadata is used to describe individual objects and how to make it simple for an audio operator to easily monitor individual objects and where they sit within the audio mix.”

Growing interest in immersive content

While suggesting that “more pressing matters” in the world may have impacted immersive audio implementation during the past few years, Lawo senior product manager audio production Christian Struck suggests that: “the immersive threads are gradually being picked up again. Tokyo, Beijing and a host of other global events are good examples of the growing interest in delivering immersive content.”

Acknowledging the variety of production approaches being taken at this point, Struck says that Lawo focuses on supporting audio engineers and the way they wish to work. He adds: “We do this by providing a physical or virtualised infrastructure that easily accommodates a sufficient number of busses and flexible monitoring options for a seamless workflow experience, [as well as by] allowing operators to control external processors directly from their Lawo console.”

Like many others, Struck alludes to the importance of sport to immersive audio’s growth as a whole, and actually implies that it could be resonating with some fans who are “probably still a bit wary to attend live events just yet [coming out of the pandemic].” Struck sees 3D audio as the missing link between 4K picture quality and an enveloping experience at home or on the go.

“I like to put it this way,” he explains. “While UHD picture quality transports crucial information more clearly, audio’s role is to convey emotions. The best way to achieve this is to create the impression that viewers always have the best seat in the house.”

The more complex productions become, the more critical is the metadata that accompanies the audio and defines how it is rendered. To this end, and as Goodman points out, the EBU has developed an Audio Definition Model (ADM) and standards for both file-based and serial metadata. This will bring greater universality to the broadcast production chain, so “content can be easily shared and re-purposed”.

• Discover more about Metadata Management

Speaking more generally, Struck agrees that a unified approach to metadata is desirable as the market develops. He says: “A highly anticipated evolutionary step could be that any given immersive production requires only one set of metadata that can be processed by both [Dolby] Atmos and MPEG-H processors, saving audio engineers a lot of work. As we progress to NGA, there is a caveat: too many options are likely to confuse end-users, which risks being counterproductive. A standard would keep content producers from investing in production time and training for deliveries that only have a few die-hard fans.”

Lawo has been focusing on its latest audio engine, the A_UHD Core, and busily “assessing every little detail regarding a future-proof immersive toolset.” The next development will be unveiled “somewhere in the fall and include more downmix options, true multi-channel channel dynamics, and all the mixing, monitoring and metering options A1s could hope for. As our A__UHD Core and mc² consoles are software-defined, this feature set will be delivered as a pure software update,” says Struck.

A gradual change in project styles

Loudspeaker manufacturer Genelec also has a long track-record of enabling immersive audio in production and post-production environments. R&D director Aki Mäkivirta implies that this is an exciting period in the adoption of immersive, with growing “freelance ability in terms of actually doing the work, and being able to select the [appropriate] gear and type of systems to produce the content.” Simultaneously, an abundance of trials and flagship event productions involving an immersive audio element confirm that “there is a lot of development in this area occurring with various broadcasters.”

Immersive-supporting consumer devices are “increasingly present on the market.”

The company’s perpetually-developing GLM loudspeaker management software has proven to be an ideal complement to evolving ideas about immersive production. Integrating with the intelligent DSP hardware within each Genelec Smart Active Monitor (SAM), the software allows each monitor and subwoofer to be networked, configured and individually calibrated for the user’s specific acoustic requirement. Now available in version 4.1, Mäkivirta says that GLM has continued to develop with “new features and capabilities that help customers to set up and run loudspeaker monitoring layouts, including immersive layouts, very effectively.”

Aware that immersive audio is increasingly being produced in a wide range of environments, Genelec has also developed a plugin, Aural ID, that utilises the individual’s unique HRTF (Head-Related Transfer Function) to create convincing binaural audio renders. Also relevant to other areas of content production such as gaming, Aural ID has received “continuous and increasing interest” since it was formally introduced at the start of the year.

Identifying audience needs

As the co-developer of one of the leading technologies in this field, MPEG-H Audio, Fraunhofer IIS is well-positioned to provide a view of where immersive audio stands now – and where it goes next. Drawing attention to multiple instrumental factors – including MPEG-H’s adoption as sole audio system for the upcoming TV 3.0 system in Brazil, and the increasingly enthusiastic interest in immersive shown by the music industry – Fraunhofer IIS head of media systems and applications Harald Fuchs notes that immersive-supporting consumer devices are “increasingly present on the market.”

From a production perspective, Fuchs indicates that a pragmatic approach which emphasises the “identification of actual needs” among an increasingly “broad range of possibilities” will remain vital. He concludes: “Broadcasters and content creators require technologies and tools that help them meet their audience’s needs, to be sure, but they also want future-proof solutions that they can integrate into their current workflows and do not require a completely new skillset.

“There are also resource concerns: businesses need realistic budgets for technologies and staff. Therefore, new developments must have the potential of being automated to a very high degree in the future to accommodate budget and time-constraints that [remain a] market reality.”

What’s coming next? Find out more about the latest advances in audio technology