Is mixed reality the future of immersive entertainment?

A new experience has pushed its way onto the immersive spectrum in the last few years. Mixed Reality - or MR - has come to describe a fully immersive experience that brings virtual objects into the real world or one which blends the physical world with the digital one.

It’s an advancement of Augmented Reality (AR) experiences like Pokémon Go - which simply add artificial elements to a real scene - but it also recognises and digitises the real world, unlike VR, which exists in a purely artificial environment.

Imagine putting on a headset and watching a football match play out on your living room floor. Or taking part in a live TV gameshow where you get to compete with contestants in the same shared virtual world.

Headset-based MR
Two main forms of entertainment experiences for MR are emerging. The first involves content producers partnering with manufacturers to create holographic experiences for high-end MR headsets such as HoloLens and the yet-to-be-released spatial computing headset from Magic Leap (see box).

Last month it was reported that British VFX company Framestore had become one of Magic Leap’s early access partners while UK agency Rewind is part of Microsoft’s highly selective HoloLens partnership programme, giving it exclusive access to its hardware and software developers kit.

Microsoft’s other partners include France’s Holoforge and Immersion and Zuhlke in Germany.

While Framestore reports that it’s “a few months away” from being able to discuss content specifics, Rewind’s first entertainment MR proof-of-concept for HoloLens, unveiled last year, focused on the Red Bull Air Race.

The real-time companion experience for the live Abu Dhabi opener created a flight deck in the user’s living room using telemetric data from the position of planes. The user was also able watch the live race with TV feeds through the headset, watching holographic planes fly overhead or being able to stand next to them, as if they were in the hangar.

For Rewind’s founder Sol Rogers - also chairman of BAFTA Immersive Entertainment - MR is “the future of live sport”. He adds: “MR works incredibly well for live experiences, especially in the area of action replays which will enhance the viewer’s perspective.” Rogers reveals that Rewind is now looking at how it can now apply MR experiences to live music concerts.

TV-based MR
The second vision of MR is not one tethered to a headset but one that’s integrated into the TV production process, exploiting advances in Chroma key technology and gaming engines such as Unreal or Unity to fuse real time graphics with interactivity in a broadcast environment.

Most recently this involved a live studio TV interview with the hologram of Belgian soccer star Eden Hazard during the World Cup. Hazard was sitting against a green screen backdrop in Russia while Dreamwall worked with broadcaster RTBT and Zero Density to provide real-time rendering through the Unreal gaming engine.

The best long form example of televised MR to date is the 8x60 minute gameshow Lost In Time, produced by Oslo-based new media company The Future Group (TFG) in partnership with X-Factor producer Fremantle Media International. The weekly prime time show, which aired last year on Discovery Networks-owned TV Norge, merged real people and objects against virtual worlds, which included an Ice Age, the Wild West and a Jurassic backdrop. An app also enabled viewers at home to play along in real time against the contestants and each other for a separate cash prize.

The multi-camera show achieved photorealistic rendering in real time thanks to a collaboration between TFG, Canadian broadcast manufacturer Ross Video and Epic Games, creators of games engine Unreal.

TFG and Ross re-wrote the Unreal code for broadcast, so that the frame rates output by the games engine’s virtual cameras and those recorded by robotic studio cameras matched.

“MR will be a slow burn, particularly while the price of the headsets remains high.” David Johnston, BBC R&D

Ratings-wise the series was a success - increasing the channel’s slot by 64%, while the mobile game was played 7m times during the season (from a local population of 5m). An off-the-shelf version of the technology behind it, Frontier [which reportedly cost $42m (£34m) to develop], was also released at NAB last year.

But creating viable business models for this type of mixed reality TV experience has been a challenge. A version of Lost in Time destined for Dubai TV was announced last year but has since been cancelled because TFG was unable to supply the deficit funding through its proposed business model – which it hoped would come from the format’s advertising potential.

The idea was that product placement could be ‘written into’ backdrop animations designed to mimic the virtual environment (a Coca Cola logo for example, could be styled to fit a saloon in the Wild West). Sponsorship could also be tailored to suit local markets.

While Marcus Brodersen, TFG’s VP of Technology still sees potential in this model he admits that TFG “couldn’t quite put together a strong enough commercial package together” for the Middle East destined format and TFG and Fremantle confirm that they have no plans to make further versions of this format.

According to Brodersen the company has now shifted its focus away from content risk and more towards technology licensing and creative services. “Doing content IP production is a high risk game and you need to prove it in the market before you can sell it,” he added.

TFG isn’t the only company revising its business strategy around MR models. Released at NAB this year, Binary Bubble’s Cortex Core content management system was initially designed to enable kids TV IP owners to bring their much-loved characters to life.

While the Unity-based system is platform agnostic, the vision was to apply it to AR, VR and MR apps so that characters such as SpongeBob could be remotely puppetered through a dashboard that relies on a number of triggers to map the character in the real world, including geolocation, object recognition and audio recognition.

According to Binary Bubbles co-founder Amit Tishler while TV networks are still part of the company’s future roadmap, for now its primary market has switched to offering YouTube influencers the means to create 2D versions of themselves and convert their archived video content into playalong games.

“The launch version of our PopBase platform is a standard 2D app so that the barrier to entry is low,” says Tishler. “We believe that social VR/AR/MR won’t pick up steam until influencers adopt these technologies and our goal is to be the driving force behind this transition,” he adds.

David Johnston, a senior product manager for Virtual and Augmented Reality in BBC R&D also believes that influencers are important early adopters - pointing out that broadcasting VR games over platforms such as Twitch, using gaming engines to composite graphics in real time, is currently one of the most popular ‘social’ uses of the medium.

TFG meanwhile is continuing to develop its MR proposition through a series of service-based partnerships. It has a running contract with the Weather Channel and recently produced an MR Tornado Explainer with viewers able to see state of the art hyper-realistic tornadoes; storms and hurricanes unfold in the studio in real-time. The Weather Channel now hopes to use MR in 80% of its programming through their partnership with TFG.

TFG has also produced segments for a live World Cup show broadcast on Chinese State broadcaster CCTV, using MR to place the studio hosts inside different cities in Russia. The hosts were also able to venture into the stadium and walk around illustrations of the players as they talked about them.

The future of entertainment?
Johnston believes that the reality of mixed reality is some way off, although he can see the BBC forming future collaborations with gaming companies and VFX houses, who have core competences in coding, 3D modeling and animation.

As far as headset-based MR experiences go - with only one holographic unit currently on the market that is primarily focused on developers – their use in entertainment is likely to be a long play. “I can see HoloLens doing interesting things in enterprise but it’s not well-suited yet for streaming video in real time,” says Johnston.

Rogers adds: “It will be a slow burn, particularly while the price of the headsets remains high. They are more likely to be used to solve business needs – in car manufacturing, architecture, medicine or even TV production – MR would allow for an infinite number of screens in an OB truck, for example.”

A self-confessed MR evangelist, Rogers predicts that cheaper, more lightweight MR glasses integrated with“ the power of AI and ML” will eventually replace all our screen based devices – which is why Microsoft is now positioning itself as an early adopter. But with headsets yet to reach a wide consumer base and the challenges of devising viable business models and use cases for longer form TV-based experiences, MR in the entertainment sphere has several hurdles to overcome before it becomes a mainstream experience, let alone one that might one day replace the smartphone.