Leaping into a new reality

A whale leaps out of the room in front of you. Penguins falls off your table. You are holding an elephant in the palm of your hand. This is mixed reality devised by Magic Leap and if you believe its head of creative strategy then these gimmicks herald the dawn of a new age of computing in which we will all share.

“Blended reality will be part of our everyday life,” John Gaeta explained to IBC365 at the View Conference for VFX, animation and immersive media, in Turin.

“So much so that we will create memories of incidents and people which are part real, part simulation. That experience will change the way we think, the way we work, how we interact socially and in turn, it will change the way artists create.”

Gaeta has led research pushing at the creative interface between human and machine for over 25 years for Microsoft, Lucasfilm and now Magic Leap, along the way devising the ground-breaking ‘bullet time’ visual effect in The Matrix. He describes himself as “a design and experience person” rather than technologist, though futurist might be more apt.

He believes the internet is about to lift into dimensionality because the technology exists to integrate believable computer-generated experiences into the world around us.

“Blended reality will be part of our everyday life. So much so that we will create memories of incidents and people which are part real, part simulation.”

Apple and Google also believe Augmented Reality to be the next stage of the internet but Magic Leap has placed the biggest bet. Google, Warner Bros. and Alibaba among other investors have pumped a mammoth $2.44 billion into the Florida-headquartered firm to build a platform for spatial computing.

“Spatial computing governs the interaction between the physical and the virtual world,” Gaeta explains. “There will be various and infinite types of digital overlays on the real world. These layers or (uni)verses will bridge the physical and the digital worlds.”

The digital overlay will be viewable by users wearing goggles, similar to Google Glass and Microsoft Hololens. What is different about Magic Leap – at least in the vision articulated by Gaeta – is the jaw-dropping scale of this connectivity which will be “full spectrum” meaning that layers can be accessed interchangeably from mobile AR to full blown head-mounted virtual reality.

He takes his cue from the immersive experiences imagined by Walt Disney in the 1950s. Alongside the Magic Kingdom which would transform animated characters into three dimensions, Disney’s original plans also included an Experimental Prototype Community of Tomorrow’ (EPCOT) intended to serve as a test bed for new city living innovations. Magic Leap can be thought of as an extension of both but on an eye-popping scale.

Beyond the theme park

“The future of Disneyland and that of every other theme park will be a hybrid of beautiful CG environments and story-based characters intermingled with physical sets, living content and performers,” Gaeta says.

Indeed, Gaeta has already worked on high concept proposals at ILM – which is owned by Disney – for “wrapping Disney World in VR”.

“Disney’s vision of immersive and blended reality will move beyond the theme park into everyday life,” he predicts. “Smart cities will be architected, built and explored by citizens using blended reality.”

Magic Leap has trademarked the term Magicverse to describe the layers which it thinks will populate the space between the real and the digital world.

While these verses can relate to any industry, sector or business from education, transportation and medicine to the military, it is the future of entertainment which exercises Gaeta. He says a new approach is needed for creating narrative content.

“Cinema and TV are built on linear storytelling and on the intuitive language of editing and cinematography,” he says.

“You don’t need to explain what a series of cuts and camera angles mean. We just comprehend it. But absent that in a direct orbiting [360-degree] view and the nature of the canvas on which you want to tell a story changes entirely. It requires a completely new grammar for communicating story.”

One consequence of this, he suggests, is that the traditional authors of a creative work will be replaced in large part by the audience itself. The “participant” or “visitor”, as Gaeta prefers to term the user experience, will shape the story simply by entering the virtual world.

“Writers, directors and other narrative artists draw on their life experience as a source of inspiration which they reframe as metaphor when they create a work of fiction,” he argues. “I think that the participatory nature of a virtual universe will mean that each visitor is their own original source such that we can collectively create a more powerful and more direct form of storytelling.”

It’s not as if Gaeta’s mixed reality world is entirely unmediated by artists or corporations, however. Everything is controlled, albeit benevolently, much like the Steve Jobs’ god-like character Halliday from Ready Player One. Again, as in Ernest Cline’s novel, Gaeta theorises that multiple virtual worlds or verses will be created as destinations for people to visit.

It’s an idea he has already spent several years developing at Disney’s immersive entertainment arm ILMxLAB as the mouse house kick-started world building around the Star Wars franchise.

“In the Star Wars universe you have countless destinations each with histories, storylines and characters that are criss-crossed all the time,” Gaeta explains. “But it is the destination which attracts people in the first place. With both VR and MR you can expose visitors to a logic of events and ongoing situations. That’s why for me it is more interesting to produce the destination than it is to control the narrative.”

He even suggests that “visitors” could play a role on the edge of some gigantic Star Wars themed VR world.

“When whole scenes or entire films are rendered photoreal using game engines then the digital universe of the story can be made accessible as a layer in the Magicverse. You could pass through any given frame like a portal and look at everything around you in perfect 3D, like a volumetric sculpture. And while you are there the backdrop would not be a static thing but dynamic reacting to your presence. You can find yourself on the periphery of these stories. You could even be dragged into the story itself.”

That’s because Magic Leap promises your own personal AI. Sensors on the headset capturing your senses (sight and sound at first, but conceivably touch, smell, and taste going forward) will help the AI understand and document your engagement with spatial computing.

“The verse will exhibit behaviours in reaction to your presence and you can interact with the environment. The product of that interaction turns out to be the story,” he says.

It is no coincidence that the Wired journalist and author of cyberpunk classic Snow Crash, Neal Stephenson, is Magic Leap’s Chief Futurist. Steven Spielberg, director of futurist fantasies Minority Report and Ready Player One is reportedly an investor.

“Spatial computing is the ability to have content persist over the world geosynchronously and in flawless volumetric 3D,” he says.

“It means the application has to be reactive to the content, to what is happening, to people who are interacting with it.

“I don’t want to sound trite. This is not going to happen in the near term.”

This involves massive data transmission, massive multi-user networking and years of work, but Gaeta thinks 5G might kick-start deployment. Specifically, in urban centres where the broadband connectivity necessary for high-fidelity realtime graphics will be fastest.

“Whole economic models will be born out of the dimensional internet,” he says. “People can socialise in verses like Second Life. They will transact through these applications.”

Not surprisingly he predicts that issues of intellectual property and user ID will generate considerable debate although blockchain may play a role in safeguarding digital personas.

“A new language will emerge as we all become a little more familiar with forms of simulated interaction,” he says. “In just a few years we have become comfortable holding face to face conversations via a screen. It is only a short step to having a holographic co-presence with us all the time.”

He is no fan of VR, blaming the failure on headset makers (like Sony and Facebook) to stimulate the home market “long before people could comprehend it”.

“VR is a car wreck,” he says. “You have to make VR first to understand it. I am concerned that VR could become isolated. On the other hand, it could be the most mind-bending computer interface. But MR is an easier path to imagine use cases emerging.”

Magic Leap’s project has critics too, having taken five years in much-hyped secrecy to get to a point where it has only just launched a headset for developers.

Gaeta defends its record, pointing out that in 2011 the start-up’s digital projections could only be viewed via a machine the size and weight of a fridge. Now all the processing is condensed into goggles not much bigger than sunglasses and a lightweight PC.

“It costs billions of dollars to have the world’s greatest engineers miniaturise this,” he says. “And it’s only generation one. Mixed Reality could go mainstream if the ergonomics are worked out and it looks cool. The point is that all development in this area has been led by older people – like me.

“We probably won’t see a step change in the technology or the media until those younger generations who’ve grown-up completely comfortable interacting by gesture and voice with a computer, take hold of the baton.”

Even so, goggles or wearable haptics of any description are surely a rudimentary means of mapping digital assets to reality. Much like The Matrix with all its attendant risk of paranoia, science may evolve to being able to program our brains directly.

“This is just a matter of time,” Gaeta believes. Indeed, researchers at the University of California, Berkeley, have used a process called Functional Magnetic Resonance Imaging to decode and reconstruct movie clips that people had already viewed just by using brain scans.

“A more advanced version of the technology could reproduce our dreams and memories in a visual format that anyone could share,” he theorises.

The concept parallels Douglas Trumbull’s 1983 sci-fi feature Brainstorm, in which scientists recorded a person’s sensations so that others could experience them.

That would bring Gaeta’s story full circle.