IBC2023 Accelerator Project: 5G Motion Capture for Performance Art and Animation

The specific challenge as stated on the project site is:

This ambitious Challenge will combine two previous Accelerator Award powerhouse winning teams under one umbrella to harness the powers of 5G indoors, this time focusing on two unique, creative workstreams that will both power live, cutting-edge motion capture and low-latency connectivity into creative output, focusing on the performing arts, and 3D animation into children’s hospitals.

• Watch the full project presentation from IBC2023 here.

The first Performing Arts workstream will use the latest in FPGA (Field Programmable Gate Array) innovations and will aim to demonstrate Ultra Low Latency 5G private networks to support the creation of new, immersive audience experiences for those at a live (or, as-live) performance/venue, and those engaging remotely in another performance/venue.

The second workstream for Live 3D Animation will aim to demonstrate a sustainable, accessible, and inclusive way to entertain children who need it most and show how IP holders can bring animated worlds to hospital wards with sick children anywhere in the world, using a system that does not require costly render farms, rely on outsourcing production, or cause studio burnout. It will also showcase how new formats of ‘live’ kid’s content can be created with meaningful, two-way interaction between the children and the animated characters.

This challenge builds on the work of 5G focussed Accelerator projects from the past two years. Malcolm Brew Senior Research Fellow, Electronic and Electrical Engineering at the University of Strathclyde who was involved in the previous two cohorts describes the primary goal of this Accelerator project as showing, “the world of production arts what ‘Pop-Up’ 5G SNPN networks (Standalone Non-Public Network) can do for them by taking what we’ve learnt over the last three years with the world’s sport’s broadcasters in delivering live Outside Broadcasts,” he said.

Roch Nakajima, President of the motion capture company Noitom International, expanded on the goal of the work, “This project aims to look at replacing both complex and expensive wired and overcrowded and unpredictable wireless connections by using 5G cellular technology,” he said. “The promise of 5G, and in this case 5G private networks, is high speed (10Gbps), low latency, large capacity, huge bandwidth, and very importantly immunity from other public wireless frequencies.”

The Problem:

Traditionally live production has involved very complex, wired rigging schemes to transmit audio, video, control signals and data. Nakajima described the challenge, “In the live entertainment industry one of the biggest bottlenecks is how to get massive amounts of data as fast as possible from one place to another,” he said.

“Note that this is a similar problem we are trying to solve in our second project which looks at transmitting mocap (motion capture) data to a monitor in a children’s hospital. Whether it is sound from a microphone, or video from a camera to an LED wall, or signals from a control board to lights - all these need to navigate a complex network of hardwired and wireless connections. This has become even more of a challenge nowadays as audiences crowd that wireless space with WIFI, cellular and Bluetooth signals, which is also an issue in hospitals.”

Brew described the advantage of using wireless systems instead of traditional methods to address this issue, “Being able to deploy a 5G Private network allows for the rapid deployment of all of the production needs for wireless audio and video feeds in venues that could not be economically rigged for live events, with issues around cable types and routing, along with health and safety considerations,” he said.

The Goal

The project is made up of two very ambitious workstreams. Nakajima explained the common goals for each, “As a motion capture company heavily involved in live events, one of our challenges is transmitting data wirelessly from our mocap suits on performers to a receiving computer which then converts that information into visual imagery on LED screens,” he said.

“We require reliable, fast, and secure connections in order for the live visuals to match the performance in real-time. The challenge we have taken on with the Strathclyde University research team is using a private 5G network to receive the data directly from the suit thereby avoiding the pitfalls of using our current 2.4Ghz network protocol.”

The second project is what the team refers to as their “IBC Accelerator for Good.” Nakajima explained, “We are looking at a different problem but one that requires the same technical capabilities. Here we are trying to deliver interactive content to cancer patients in airborne infection isolation wards in children’s hospitals,” he said.

“Again, the challenge is getting a high-quality bi-directional signal from two distant locations to deliver interactive animated content to children undergoing treatment. The added dimension to this project is to also look at the concept of customized interactive broadcasting and how to make it inclusive no matter what the situation is. In this case children who may be separated from most human contact for extended periods of time.”

Brew addressed the technological challenges the project is tackling, “We’re having a crack at building an immersive multi-venue experience by taking control of multi-channel media and signalling streams, helping manage their timely arrival for synchronisation, and helping to merge all venues into one shared performance where the remote audiences can be immersed in a shared multi-venue experience, he said.

He emphasised the advantage of doing this within the Accelerator programme which has given them access to the “latest ‘Billy Whizz’ FPGA’s that enable both the encoding and decoding of 4K UHD video streams at super low latencies of sub 60ms, as well as being used in generating the Software Defined Radio 5G NR waveforms,” he said. They have also been given access to the latest in high framerate laser projection systems for the project.

Participants

IBC’s Accelerator programme is unique in the industry for creating an environment that brings together multiple companies and individuals with a broad range of skills.

“On the technical side Malcolm Brew and his team at Strathclyde University are central to this project,” said Nakajima. “Their expertise is second to none in the field of 5G private networks. We also have the support of AMD, AT&T and Vodaphone for hardware and transmission logistics. In our second project, we are working with Grainne Owen’s organisation - Curing Kids Cancer - as well as several Children’s Hospitals. They have put a human face on technology and driven us to develop the project, not just this year, but well into the future.”

The Accelerator Process

IBC’s Accelerator programme is unique in the industry for providing a safe place for a variety of companies to come together and collaborate on solutions that benefit the entire industry as well as providing access to cutting edge technologies and equipment.

Nakajima feels that a project like this could not have been done without the Accelerator programme: “Working with this diverse group of experts is not something a small company can do in-house. It allows us to tap into the knowledge of professionals we would normally never have the chance to work with,” he said.

“It also allows all of us to develop case studies that normally would have little chance of coming to fruition as they are not profit driven. The accelerator program is unique in that it brings experts from a wide spectrum of fields who normally might not have had the opportunity to work together. In this case we have engineers working with hardware manufacturers along with artists and non-profit agencies. And most importantly connections with amazing minds who have become friends over the last three years.”

“We wouldn’t have had any of the bleeding-edge multi-channel immersive audio equipment and expertise, or low latency laser projectors without IBC involvement,” added Brew. “Lots of what we’re doing in this year’s Accelerator has been handed on from all our interactions and experiences via previous collaborations with the BBC, OBS, Haivision, Sony, and Warner Brothers, and has been recognised with various industry awards.”

The Showcase at IBC

Each Accelerator programme culminates in a showcase at IBC2023 in September. Catch the 5G Motion Capture For Live Performance And Animation project, showcasing its Proof of Concept and findings at IBC2023, Friday 15th of September, from 11:15-12:15 at the Innovation Stage.

The team are planning to pilot test broadcasts directly with hospital administrators and art therapists, and ultimately with the children themselves during their presentation.

Benefits

Nakajima sees multiple benefits coming from being a part of the Accelerator programme, and to just for the participating companies, but for the wider industry and ultimately the audience. “The live entertainment event industry is a completely new market for this,” he said. “The challenge is transmitting multiple types of data standards within the same 5G private network and not only making sure the data gets there fast, but also gets to the correct destination. What the team at Strathclyde University is attempting is to make ‘live’ as live as possible - bringing motion capture data, music, and video together in a performance venue. We take that one step further with the second project where we try to cater to and deliver content to a very small and unique audience. Here we leverage the technology to bring smiles (and therapy) in real-time via an animated character to small children undergoing cancer treatment.”

Read more IBC Accelerator Programme