Virtual Reality in the frame: what works and what doesn’t for 360-degree video

For broadcasters the real challenges are in drama, where the conundrum is how to offer an immersive and even interactive experience while still telling a story. For live production, the biggest issues are technical.

For live productions, the ability to offer a ‘you are there’ experience is the main draw for 360-degree video. VR views, whether the occasional whole game or highlights, are now available for a wide range of sports, including weekly NBA basketball games.

VR cameras are being set up above the centre circle or behind goals, or on racing cars, providing a new perspective on the action. Some sports are being rather tentative in their experiments, creating only a CG virtual environment in which to watch conventional 2D views (as happened with this year’s Super Bowl), but as VR usage increases, expect more events to be made available in 360-degree video.

“Stereo in VR for live action video sucks, really bad. I would strongly recommend that people avoid trying to do stereo VR anytime soon” – Alx Klive, 360 Designs

There are a variety of immersive content types, from straightforward 360-degree video to true VR creating artificial environments using gaming engines.

For broadcasters, “360-degree VR is very exciting, and at the moment we’re trying to work out the cinematography of it, the storytelling,” said Alia Sheikh, Senior Development Producer, BBC R&D.

“What happens when the viewer can not only look around everywhere, but that the whole point of the experience is that the viewer feels they have that control? You can’t really take it away, so how do you direct attention, how do you lead someone through a story?”

BBC R&D has been experimenting with theatre techniques and presented a paper at IBC last year.

Where, previously, VR had been driven by its use in marketing, “now narrative, technique and art are becoming more important,” according to Alaster Armitage-Brain, VR and Interactive producer, Happy Finish.

“The narratives have changed in VR as the technology has advanced,” added James Manisty VR Director, Alchemy VR, Atlantic Productions, who specialises in natural history.

A problem he has seen with trying to produce narrative VR productions is that once the ball rig is put in position, and you have hidden behind a wall or tree, it is difficult to know if it is recording correctly. “Only to come back when your carefully scripted scene has completely failed because one of those four or 16 cameras has gone down.”

Drawing attention without causing whiplash

Producers of 360-degree video should remember that people generally want to look ahead through about 180-degrees in front of them, and not to try to have lots happening in every direction, warned Tanya Laird, Founder & CEO, Digital Jam.

“There is this real desire at the moment from people who are creating 360 content to literally give their audience whiplash,” she said.

“We’re talking about an experience that still needs to be human. We don’t have a neck that swivels in 360-degree, so we don’t need to keep putting stuff behind us. Just because it’s there doesn’t mean we necessarily need to use it.”

DoP Robert Hollingworth, who has worked on immersive 3D IMAX productions, agreed: “With a good story going on in front of you that’s filling your vision, you don’t need to look behind. There is this idea with 360 that you have to be looking behind the whole time – I disagree with that. If you’ve got something interesting in front of you, you’re not staring behind you.”

Of course, there should be something there if people look, the picture continues just as it does in the real world, but it is only really the first time that people try VR that they are continually looking all around them, but once they realise there is a story going on, they will focus on that.

Typically, viewers of VR “want to go somewhere you can’t go normally, so you want to give someone an experience they otherwise can’t get,” such as underwater, said Hollingworth, who also thinks that “VR for education is fantastic.”

“You could completely lose a viewer through poor shot selection or not directing the viewer properly,” said Manisty.

“Things like sound and, in particular, light are cues that are really useful for content producers to make cuts in VR, because you can’t do what you normally do in TV, cut to person A, cut to person B, you have to make the viewer choose that themself.”

Hollingworth believes that productions should use off-speed techniques wherever possible.

“I think shooting anything on speed, unless there’s sync sound is pointless. You’ve got to take someone where they are not going to go, so let’s pop them in a slow-motion world, a timelapse world, lets bind those timeframes together with some compositing. That, for me, is the exciting side of things.”

However, finding a camera that does high-resolution slow motion is tricky. For timelapse, three Nikon D810s with Sigma lenses will give “an enormously high resolution” shot, he said.

Lighting the way by trying to create shadows

“In daylight, everything just works, the sun is really bright, so you’ve got to worry about your shadows,” explained Sheikh.

“Once things get dark, the temptation is to fix that, make them bright, but actually when you go in to a dark environment, it’s already quite immersive.”

BBC R&D did a shoot in a mine, and lit up the set with lots of little lights either replacing an existing light with a better one, or hiding them - lighting behind pillars. “We were trying to create shadows – that’s what provided the depth.”

In another shoot, where there was a girl wandering around a very busy warehouse they gave her a lantern to hold so that viewers would be drawn to her.

There is also the problem of getting viewers to look at an event when it happens, and for this they drew on techniques devised for theatre, which “has a lot of experience in leading the audience’s direction around a space that doesn’t move,” such as: having the character who is already the focus of attention look at the new thing, or have something cut across the audience’s field of vision, or have the new thing make a noise - combinations of these work particularly well.

“In a dark environment, even though it’s not great to film in, lighting suddenly becomes your friend because you can physically move the light and you don’t want people to feel as if they are being told where to look, but it’s a very natural thing – you look where the light is,” said Sheikh.

Mono versus stereo

Some VR cameras can do stereo 3D, but “stereo in VR for live action video sucks, really bad. I would strongly recommend that people avoid trying to do stereo VR anytime soon,” said Alx Klive, CEO, 360 Designs, which specialises in live VR.

It is “extremely hard to work with; you’re going to spend three or four times as long in post to do stereo than you will with mono.”

You have to stitch both the left eye and right eye, and if you have any issues, such as a shake or needing to do a horizon adjustment – if you do rotoscoping on one you have to do it on the other. “There is no such thing as true stereo VR right now for video,” added Klive.

You can have it in one axis (horizontal), so viewers can look around in stereo, but if they tilt their head, the stereo disappears. “That is down to an architectural issue that is really hard to solve,” and any solution is at least five years away - as it would require all the stitching to be done in the headset, “which will require crazy computing power to do.” Besides, nausea and discomfort are much more pronounced with stereo VR.

Also, as stereo requires two streams, for any given bandwidth the video will be half the quality, and as the delivery medium is generally a mobile phone, there are bitrate limitations, therefor the quality will look better in mono. There are also fewer tools available for stereo VR, and more workflow limitations. Distribution also has to be taken into account. At present, most VR is delivered via YouTube or Facebook, and most people are experiencing it either in a browser (and scrolling around) or on a phone/tablet and tilting and swiping, neither of which are stereo.

Besides, “viewers don’t really notice.” If anything, “people prefer great mono. It just looks better than poor stereo,” Klive told a seminar at BVE in London.

If you do still want to deliver stereo, “you are much better off to shoot mono and do a stereo conversion. That is how the few people at the top of the industry who are doing it, like Felix & Paul Studios are doing it right now.”