Framestore’s CTO Lincoln Wallen says the creative industries should acknowledge the art of math to craft new interactive AI-driven social experiences.
The future of media and entertainment will be interactive, social, and fuelled by AI with companies versed in blending tech and art at the forefront of development, according to Framestore’s Chief Technology Officer (CTO), Lincoln Wallen.
“It’s going to be a slow burn but the technological basis of it has already been built,” he says. “Imagine experiences in which consumers can interact but are nevertheless driven by narrative visual storytelling. It’s a mixture of active lean forward and passive lean back. That is a media type that I expect to explode over this next period.”
Wallen joined Framestore and Company 3 Group (FC3) last summer from Improbable, a tech developer building platforms for multiplayer games and battlefield simulations which was valued at $3bn in 2022. He previously spent four years at Electronic Arts building its mobile games business before entering the film industry at DreamWorks, initially as Head of R&D from 2008, and then promoted to CTO in 2012.
Read more Framestore Company 3 hires Group CTO
That career journey has given Wallen an orbital perspective of future media in which technology and content, both IRL location-based and virtual, merge to create new forms of real-time communication and storytelling.
“The metaverse will come back as all truly disruptive ideas do” Lincoln Wallen, Framestore
“If you want to spot the metaverse as it emerges it’s going to come in interactive experiences in which large numbers of consumers come together,” he explains. “We do this today in social environments when out shopping, at sporting events, on the street, at festivals and conferences, but the digital world doesn’t properly support that sense of social engagement.
“The only markets in which people really come together digitally are video conferencing and video games but those experiences are designed for very specific numbers of players, not for crowds. Being able to interact with people meaningfully in a digital environment changes the game.”
Technology that facilitates simultaneous multi-player interactions at scale will underlie future media says Wallen, who believes this has been cracked by Improbable (of which he remains a Non-Executive Director). The next step is to evolve the user experience.
“VR really hasn’t left the laboratory in terms of the consumer experience. It still isolates the user. But the metaverse will come back as all truly disruptive ideas do,” he says.
An example of what he means is the work Improbable has done with Major League Baseball (MLB). Here baseball fans can gather in an MLB virtual ballpark and interact simultaneously in a single place, at a naturalistic scale. It claims to be the first service to host such large interactive sports experiences.
Another example includes fashion brands that might organise a virtual runway show inviting people to the physical catwalk and thousands more people to virtually inhabit the same space. Wallen believes this type of experience will eventually happen daily as physical interfaces like mixed reality goggles improve, but that the content itself won’t work unless crafted with storytelling expertise.
“The audiovisual world of linear media has built up an enormous bank of techniques creating adverts, TV shows or movies to grab a consumers’ attention and to stimulate emotions but those techniques are not used today within interactive environments,” he insists.
“Those working in linear media will call it ‘immersive’ as a way of trying to signal that this is something you’ve seen in VR experiences or in location-based entertainment. What we haven’t yet seen is that experience realised in a fully digital environment. It’s neither gaming nor linear but something in between. The potential of this middle ground is enormous.”
“The linearised model of film production is going to collapse into a much more iterative holistic model” Lincoln Wallen, Framestore
If Hollywood studios are too entrenched in their own structures to pivot fast enough, companies that are operating across market sectors will be able to synthesise these capabilities and exploit them. Framestore is one. Its work ranges from the VFX on Gladiator II to designing an interactive exhibit for the Science Museum.
“That isn’t to say that Framestore is suddenly going to become a content company,” Wallen stresses. “It means identifying those brands and IPs that are going to pioneer this new media and serve their existing customers in a different way.”
Framestore launched in 1986 at the birth of the computer graphics revolution. It remains headquartered in Soho where it has navigated and led the industry-wide transformation of post-production from an afterthought into an integral partner spanning the creative industries.
“The linearised model of film production is going to collapse into a much more iterative holistic model,” Wallen says. “Right now, the previz phase and the production phase feel separate. I think AI is going to bring the back end right up to the front entrance and make certain complex processes real-time.”
Bridging the uncanny valley
For Wallen, the key to the future of interactive media is the ability to create digital representations that talk, behave, respond and ‘think’ like a human. Such avatars could be our own online personas or the embodiment of a corporation. AI is already supercharging such developments in video games where AI-driven non-player characters (NPCs) are being introduced to interact in a more lifelike manner with game players.
“You are going to see NPCs having existential dialogue with game players,” Wallen says. “This is one of the most exciting things to me. Large language models (LLMs) have exposed a degree of structure in natural language that makes dialogue possible at a level that it hasn’t been in the past.
“When people look back on this era it is going to be the impact on language that will be seen as the most profound. Natural language will become a much more effective interface to [digital] systems and this will make them more accessible, which will be an enormous economic benefit.”
There is a problem though. The increasing sophistication of LLMs must be matched by sophistication in visual behaviour, emotion and characterisation. The uncanny valley is yet to be bridged.
“We don’t have a language of performance. There’s nothing to correlate spoken language and non-verbal gestures to make your NPC seem alive. Animators study human behaviour and know exactly how to produce that effect but we don’t have tools that do that automatically. You can’t take the animator out of the equation because they understand what human gestures mean, but the way they’re created right now is painstakingly frame by frame. We have to build that capability.”
Read more Synthetic-based humans: a solution to talent shortages or threat to skilled jobs? | Industry Trends
The EU-funded research project PRESENT (Photoreal REaltime Sentient ENTity) of which Framestore is a partner aims to create virtual sentient agents that look and behave entirely naturalistically. The aims are to demonstrate emotional sensitivity, establish an engaging dialogue, add sense to the experience, and act as trustworthy guardians and guides in the interfaces for AR, VR, and more traditional forms of media.
“It’s only the high-end media industries that actually know how to produce digital performances, that that make people feel that NPCs are alive,” Wallen says.
Understanding how the world works
“There’s a very ambivalent relationship in the humanities between systematisation and what they think the humanities are all about, which is inspiration” Lincoln Wallen, Framestore
Wallen has a PhD in artificial intelligence from the University of Edinburgh, and subsequently taught at the Oxford University Computing Laboratory before moving into the media industry. He is also a non-exec board member of Finnish VR/MR headset maker Varjo, Pearson - the FTSE100 global education company, an advisor to bp and a 25-year member of the maths and AI strategic consultancy Smith Institute. Few people understand the intersection of science with the humanities as well as he and that means his view on AI as an artistic tool is not binary.
“All through my career the driver has been the interplay between modelling the world and activating it through computational systems,” he says. “Understanding how the world works is a passion.”
What strikes him about the current AI wave is the tension it has caused between inspiration and predictive calculation, but it’s a tension that is not new, and the two are not mutually exclusive.
“Making movies is a balance between algorithmic processes and artistic choice. That’s a very natural combination that AI is forcing to be recognised.”
He explains that neural networks, a mathematical construct designed to mimic human thought, has reached a level of maturity that demonstrates structure in natural language.
“We now understand that there’s an awful lot of structure in writing. It’s not just a grammatical structure. Literary critics have argued this for centuries, but the humanities oscillate between seeing these patterns and then running away from them because it seems to be contradictory to creativity and choice. There’s a very ambivalent relationship in the humanities between systematisation and what they think the humanities are all about, which is inspiration.”
He argues that there is an art to mechanisation: designing-in specific outcomes that preserve human creative agency, with plenty of scope for machine learning technology.
“LLMs don’t manipulate intent,” he insists. “Humans create intent. They communicate and collaborate to organise their efforts to produce fantastic outputs, whether that’s a video game or a film. Without human dialogue, without the imagination that’s really framing the outcomes, you don’t get the products.”
We can use advanced AI techniques to try to realise that imagination, he says.
“Computational techniques are forcing recognition that there are structures, including within narrative visual storytelling.”
Watch more AI - Episode 3: Gen AI in practice - deep dive on deep fakes
Language of intent
Wallen thinks the industry needs a standardised way for artists and filmmakers to communicate with each other in the age of AI, using AI tools. Current-generation AI tools, he says, are not intuitive and actually hinder artistic collaboration.
“GenAI tools work at a very low level. They don’t help you express what you’re trying to do. The only way to get the machine to respond is to give it instructions bit by bit of what you want to see but not why you want to see it.
“What we want is a language of intent,” he argues. “We want what the artist is trying to achieve made more directly expressible and reflected by the interfaces of these tools. If we had these languages we would be able to build technologies that really turbocharged people’s creative capabilities. That I think is what the end result of this AI wave will be but only if the creative industries start looking at their systemisation techniques and their content [Wallen uses the term folklore) and turning it into activatable language.”
This is something that Wallen is exploring at Framestore but it’s too vast a challenge for one company alone.
“I’d hesitate to say that this is a century of development but it’s also not easy. What I’m trying to do is stimulate academics and creatives to think about the problem. Even small areas of advancement in this area can lead to enormous benefits to the creative process. It is one of my personal motivations for joining Framestore.
“Whether it’s your digital proxy, virtual playmates, NPCs in games, a retail concierge, or part of the crowd in a virtual stadium doesn’t matter. There are so many applications of LLMs that are going to explode when coupled with character.”
Wallen completed a thesis on Mechanisation in Mathematical Art during the 1980s and suggests that the challenge is as old as the Scientific Revolution itself. He refers to early computers which worked by punching sequences into cards.
“By attending to the design of, and access to, the tools with which we ‘punch the cards’ today we can preserve the poets and even pursue ‘poetical science,’” he says.
Read more VFX At The Edge: Adapting to the unpredictable challenges of modern production featuring Framestore
No comments yet