IBC2023 Accelerator Project: Synthetic Humans

The Challenge

The specific challenge as stated on the project site is:

The aim of this Accelerator is to present two different use cases based on synthetic humans: the first one is a theatre filled with the melodic tones of Maria Callas, and the second is a photorealistic sign language interpreter, to address important aspects of accessibility in broadcasting. These two distinct use cases will aim to demonstrate how synthetic humans can be used to captivate audiences in visually stunning, emotionally moving, and inclusive ways.

• Watch the Accelerator project presentations at IBC2023 here.

The Problem

Roberto Iacoviello, Lead Research Engineer at RAI, held the responsibility of overseeing Work Stream 1, which the team coined as ‘Entertainment.’ Michael Davey, Technology and Operations Director and Co-Founder of Robotica Machine Learning, took the lead on workstream 2, the ‘Accessibility’ workstream.

Iacoviello referred to their synthetic humans as ‘digital twins.’ Building a digital twin comes with its own set of challenges and issues. Some of these problems include:

1. Data Integration and Quality: Creating an accurate digital twin requires integrating data from various sources, including motion suits, camera mobile, various software, and existing archives. Ensuring the quality, consistency, and reliability of this data can be a significant challenge, as data may be noisy, incomplete, or inconsistent.
2. Interoperability: Different components of the workflow may use diverse technologies and standards, making it challenging to create an integrated pipeline to build a digital twin. Ensuring that all components communicate effectively and seamlessly can be a daunting task.
3. Model Validation: Developing accurate 3D models that closely mimic the behaviour of the humans is essential. Validating these models to match real-world performance in terms of resemblance and movements can be difficult, because of the lack of metrics or standards.
4. Cost and Resource Allocation: Developing a high-fidelity digital twin can be resource-intensive in terms of both time, money and people involved in the project. Leveraging AI-based software and tools can be a strategic challenge.
5. Lack of Standardisation: The lack of standardised frameworks and protocols for developing digital twins can lead to compatibility issues and difficulties in collaborating across different organisations or industries.
6. Cultural and Organisational Resistance: Introducing new technologies to build digital twins might require changes in how production centres work and make decisions. There could be lack of skills or resistance from employees who are unfamiliar with the technology or reluctant to adopt new processes.
7. Commercial: The lack of standardised frameworks and protocols for remunerating talent for work their digital twin undertakes, and for assuring the quality of their work creates barriers and risk.

The Goal

The overarching goal of the project is to explore a variety of 3D production techniques and different motion capture technology tools to build synthetic humans that can accurately and realistically replicate human movements, facial expressions, and voice to create and publish more believable and realistic characters.

“Work Stream 1 had the ambitious goal of resurrecting historical figures and personalities from the past,” said Iacoviello. “Our collective efforts have been aimed at leveraging the latest technologies to bridge the time gap and reintroduce these significant individuals into the present. Through research, advanced simulations, and innovative 3D model building, we seek to provide a captivating experience that not only educates but also entertains, allowing contemporary audiences to engage with the past in a new and immersive manner.”

“We are looking at presenting weather forecasts in British Sign Language for Workstream 2” said Davey. “Documenting UX approaches to toggling the signer on and off analogous to how you toggle subtitles on and off and developing a contractual framework for remunerating BSL talent.”

The primary areas of focus within both workstreams are:

• Representing people as photorealistic 3D models
• Real-time capture of body and face
• Lipsync technology optimised for speech and singing
• Leveraging Broadcaster’s archive content
• Review cost-effective, sustainable tools
• Output on multiple devices (VR headset, mobile, etc.)

The final Proof of Concept will aim to showcase how synthetic humans can increase inclusivity and allow endless storytelling possibilities within emerging platforms, and be used to enhance traditional media, such as television programs, live on-air presenting, and broadcasting.

Participants

IBC’s Accelerator programme creates an environment that brings together multiple companies and individuals with a broad range of skills.

Iacoviello and Davey highlighted several advantages to be found in working the in the cooperative environment enabled by the Accelerator structure.

Collaborating with other organisations offers a range of benefits that extend beyond what can be achieved solely in-house. Access to new technologies, sharing of learnings, exploration of new use cases, and collaborative research all contribute to enhanced innovation, efficiency, and competitiveness in an ever-evolving business landscape.

Partnering with external organisations often grants access to cutting-edge technologies and tools that might be beyond the scope of an in-house team’s expertise or resources.

Working with different organisations exposes the team to diverse perspectives, methodologies, and best practices. This sharing of knowledge fosters a cross-pollination of ideas, enabling you to adopt solutions that have proven successful elsewhere.

Collaborative efforts often introduce new use cases that individual organisations might not have otherwise considered. External partners bring fresh viewpoints and experiences, which can spark creative ideas and open doors to unexplored markets or TV formats.

Partnering with external organisations can facilitate joint research and development initiatives. This can be particularly advantageous when tackling complex problems that require specialised expertise.

The Accelerator Process

IBC’s Accelerator programme is unique in the industry for providing a safe place for a variety of companies, sometimes competitors, to come together and collaborate on solutions that benefit the entire industry.

Davey was enthused by the programme. “This was my first IBC Accelerator, and I’ve thoroughly enjoyed it,” he said. “I would relish the opportunity to participate in future accelerators.”

Iacoviello liked having sight of the other projects. “We have noticed that there are other interesting Accelerators,” he said. “And it would be a good idea to merge with them doubling the outcomes. For example, the accelerator working on 5G can provide live 3D animations.”

The Showcase at IBC

Each Accelerator programme culminates in a showcase on the Innovation Stage at IBC2023 in September. Catch the Synthetic Humans For Entertainment And Accessibility project, showcasing its Proof of Concept and findings at IBC2023, Sunday 17th of September, from 13:30-14:30 at the Innovation Stage.

Visitors to the Innovation Stage and expect to see two different use cases based on synthetic humans. |The first is a theatre filled with the melodic tones of Maria Callas. The second will feature an avatar representing a real-life deaf presenter who communicates through sign language. This avatar utilises voice synthesis to vocalise what the presenter is signing to address important aspects of inclusivity in broadcasting. These two distinct use cases will aim to demonstrate how synthetic humans can be used to captivate audiences in visually stunning, emotionally moving, and inclusive ways.

Benefits

The team identified a number of important benefits to be realised by this project.

Cost-effective tools

In order to guarantee the quality of the reconstructed 3D model, the traditional reconstruction pipeline relies mostly on large and expensive acquisition systems that significantly limit the wider application of human reconstruction. These problems have motivated the search for low-cost, learning-based approaches that aim to reconstruct 3D humans from only a single or sparse-view 2D images. In addition, the use of audio-visual input to drive the synthetic human movements allows for a reduction in the time spent animating keyframes.

New workflows

Breaking news, economy, elections, sports, and the weather forecast are just a few of the wide range of topics that are covered by broadcasters. Showing these topics with an understandable and visually pleasing approach is the way to ensure the interest and involvement of the audience. Synthetic humans combined with game engines have the potential to give broadcasters more creative freedom, a new way of working, and the promise of accelerating traditional pipelines.

Previsualisation

Traditional production needs to adapt to the collaborative aspect and to work with interactive non-linear event plotting. The director has to plan the XR workflow before he even arrives on set, mapping out the space and pre-visualising the appearance of each virtual asset at the rendering stage, taking into account new platforms and new devices.

Using near real-time 3D reconstruction tools and game engines, pre-visualisation will allow directors to ’see’ productions before TV programme is proposed and budgeted. This leads to the exploration of new collaborative workflows between the department heads who, working as a team, can understand the advantages and disadvantages of the various tools, as well as the 3D model generation technology and game engines currently available.

Archive exploitation

The most recent AI tools and game engines can be combined with broadcasters’ archives to produce synthetic humans. This will enable new ways to make use of the huge amount of audio-visual content that has already been produced and archived.

Read more IBC2023 Accelerators in full