IBC2024 Accelerator Project: AI Media Production Lab

No comments

Three project streams explore the practicality of using AI in audience validation, creative storytelling, and live sports and event production. Mark Mayne reports.

The exciting and forward-looking umbrella theme of ‘AI Media Production Lab’ is a particularly topical area for the IBC2024 Accelerators Programme. The three streams that make up this project tackle some of the most technical and complex data problems facing the media industry today, and seek to throw light on the future of AI in the media and improve inclusion and diversity - a truly impressive and disruptive mix.

AI Audience Validation Assistant Project - the pitch

The AI Audience Validation Assistant (AAVA) Project was proposed by Champion Zwart and Evangelische Omroep (EO) and supported by Co-Champions BNN VARA, Channel 4 and YLE. The central plank of the project is to use AI as a tool to combat bias while promoting inclusion and diversity by developing a broad range of AI personas that represent the complexity of today’s society. These AI personas will then be able to guide media organisations to create more engaging content, being available at an early production and development stage without the cost and complexity of current methods such as focus groups.

O’Jay Medina, Data Engineer at Zwart, has been in the driving seat, and says that a strong start has been made to a complex and challenging project: “The main building blocks of the project are about finding, structuring and validating the data sources that we need for our digital twins. With this input, we can continue with the AI engineering part to generate content feedback from our target audience personas. Simultaneously we are thinking about the concept and adoption of our showcase, such as what type of content, what type of persona, and what type of feedback [will be interesting for the IBC2024 audience].

“We are talking with parties that can help us make the decision on what type of data to use to generate the most optimal digital twins for this specific use case”, adds Medina.

The ultimate endgame would be to create a standardised interface to query the personas: “Another option would be to select or filter characteristics which eventually result in one or more multiple digital twins (personas), or a more sophisticated way would be of course to give users the ability to ask questions about/describe a specific persona.”

The digital twins will also need some element of real-world input to stay current, as Medina points out: “Ideally, the configuration will be built in such a way it leverages a dynamic/live data source, [such as to] give it the ability, for example, to use data sources like news websites to make sure that the digital twins stay up to date.”

AAVA Project - challenges

There are significant challenges for the AAVA Project, not least on the data side. “We are still in the phase of deciding on what type of data to use and from which sources,” says Medina. “Also [we are still deciding on] the input datasets - demographics are an option but we could of course also choose to use focus group data (that we as broadcasters already have access to).”

IBC_Panel

IBC2024: Accelerator kickstart day panel

Once that has been decided, there are still several hurdles to overcome. “Ingesting the data will represent a challenge too - not necessarily from the technical perspective but more from the semantical side - how do we make sure that the multiple data sources together provide value towards potential digital twin feedback?” states Medina. “We make the regular joke that we wish we had digital twins ourselves to progress this project faster, which appears to have several challenging areas!”

However, progress has been rapid, and learnings have already been extrapolated: “So far we have talked to a couple of universities and research groups which I have enjoyed a lot, my biggest learning so far is that media companies should leverage/engage more with educational/research institutions to share knowledge and expertise, especially when it comes to data and AI,” he says.

Overall, Medina is upbeat on the future of the project and AI: “I think AI offers a lot to the media industry: enhanced content personalization, efficient data analysis, and new interactive experiences, while in media production, it may offer considerably streamlined processes such as editing, scriptwriting, etc. In the near future, AI’s role will likely expand even more, leading to more dynamic and immersive content creation, as well as highly personalised audience experiences.”

Generative AI in Action – the pitch

Proposed by Champions Rai and EBU, this project aims to leverage AI in creative storytelling and production by using a range of AI tools to make a pilot with the help of participants Pluxbox, Plan IX Labs, Respeecher, Xansr Media, Infuse Video, RKG Group and Somersault Agency.

IBC_Pitch

IBC2024: Accelerator kickstart day

Roberto Iacoviello, Lead Research Engineer, Rai has been masterminding the ambitious project. As he summarised in his initial pitch: “This project will investigate the impact of AI in the broadcast environment to produce an animated and live-action TV series, leveraging AI for the script development, starting from an input by creatives, then visual storyboarding and finally multimedia production, such as 3D images, audio and music, with the aim of increasing workflow automation, efficiency and innovation.”

A veteran of the IBC accelerator programme, Iacoviello is building upon two previous weighty projects, 2021’s Smart Remote Production for Real Time Animation, and last year’s Synthetic Humans for Entertainment and Accessibility.

“The innovation here is that we don’t want to use just one tool,” he explained. “We want to lay out the entire broadcast pipeline based on generative AI, and this is not only going to change how we produce content, but also what stories we can tell and how we interact with our audience.”

Generative AI in Action – challenges

As Iacoviello pointed out, the project will not only explore the limits of existing generative AI technologies across a host of major platforms, but also benchmark it: “The key lies in understanding the balance between automation and human creativity and the critical question is whether the quality achieved through generative AI meets the standards expected for broadcast content.”

Another key question will be interoperability, as the different tools will inevitably have differing data structures and syntax, and differing output formats which will pose challenges for the project.

“We will also be comparing the traditional workflow with the new AI-based one, and while, probably, the ultimate answer will be that the creative touch of a human professional is still irreplaceable, it remains an interesting question,” confirmed Iacoviello.

Changing the Game: Predictive Generative AI - the pitch

Last but by no means least, ErinRose Widner, Global Head of Business Strategy, Emerging and Creative Technologies - Media & Entertainment, Verizon Business, has been spearheading a project that looks at AI as a production tool for live sports and live events to assist directors and production teams to potentially predict the next moves for swifter edits, content detection, and personalised predictive interactivity. The challenge was proposed by Verizon Business, and includes Champions Paramount Global and Al Jazeera, and Participants Xansr Media and Magnifi.

At the IBC2024 Accelerator Kickstarter day, Widner was keen from the outset to emphasise that humanity and authenticity create the core of this proposal, a stance that has been successfully carried into the workflow: “Authenticity is at the core of the architecture we are building out now. Not only are we looking to delve into a person’s preferences (based on the information they provide), we are currently also planning to have a calibration element in the workflow to ensure that the information we plan to tailor feels right to the viewer.

ErinRoseWidner_Pic_2022_Cropped

ErinRose Widner, Verizon Business

“The goal is to take a piece of live content that can be personalised in real-time, using the preferences that we have gathered at the beginning from a viewer and maximising low-latency connectivity to ensure it feels like an authentic interaction. The biggest challenge with this project has been the content itself, as we need permission for whatever we use. So depending on permissions, we may have to create our own content or use a piece of content in a way that mimics a live playout.”

Changing the Game: Predictive Generative AI - challenges

Indeed, content has proven to be one of the central challenges according to Widner: “We would ideally love to use a team sport, but we may end up looking at an individual sport or shooting our own content to prove out this experience.” She’s also keen to point out that the sheer scope of the benefits of predictive generative AI in live means that there’s no shortage of options: “In brainstorming content that we could use, we’ve really run the gamut in ideas from football (soccer) to cricket, to kabaddi, to freestyle football, to skateboarding, to sailing, to cycling, which just really speaks to how many sports and fans could benefit from this kind of unique personalization.

“The broadcast industry is evolving. It’s not just linear anymore, so much content is now being created and consumed for different types of connected devices and a variety of platforms, and so much of that content is live. So when you start to look at the full picture, content, consumption preferences, connectivity, along with technology currently available (and anticipated), we begin to see that next era of entertainment being shaped to an incredibly exciting new experience.”

Widner took a closing beat to broaden the context, mulling the value of innovation to the entertainment and media industry.

“What’s funny is that AI has been such a big buzzword for the past year or two, but when you think about it, there has already been so much AI around us for a while now. While it’s true that AI may make some jobs easier or redundant, technology has also always created new opportunities for the entertainment industry.

“I’ve thought a lot about how people were fearful when we stopped shooting on film and went digital, and if you go even further back and think about going from silent films to talkies to colour, you can see how each of those iterations of entertainment brought some trepidation around change. But it also unlocked some magic in what we were able to create. AI is just another technology tool in our entertainment evolution.

“That said, I am very aware that we also need to be mindful and thoughtful of how it can be used, but I am excited about the potential as long as we can be ethical and maintain human authenticity. Because at the end of the day, entertainment that truly resonates with audiences, is content that connects to you as a person, on a very human level”, she concluded.

All three workstrands will be presented and demoed in full at IBC2024.

The list of companies as Champions and Participants across the three project streams includes:

Zwart

EO (Evangelische Omroep)

EBU

Verizon Business

ITV

Channel 4

YLE

BNNVARA

Pluxbox

PLAN IX LABS