Spotlight on: Immersive film makers

Jannicke Mikkelsen

Jannicke Mikkelsen is a VR film director and cinematographer based in London. Mikkelsen worked as a stereographer on Paramount Pictures Hansel and Gretel: Witch Hunters and was eager to pursue the creative opportunities as a cinematographer. She was director and 360 stereographer on British rock band Queen’s 2017 film VR The Champions and worked on Norwegian television series 71 Degrees North and David Attenborough’s Great Barrier Reef.

As an immersive cinematographer what is your primary job function?
Our role is to tell a story, and I find documentary to be a truly immersive experience because you can plunge the viewer into the environment and allow them to discover things. The opportunities are endless, making immersive a really great experience for viewers.

As a creative and technical cinematographer from the filming point of view it remains quite traditional following the context of a story narrative. The challenge with immersive and interactive storytelling is you can’t control where the viewer will choose to go but you can guide them through your thought process, so it’s really important you are smart about how you expose your viewer to the narrative.

Virtual reality (VR) and augmented reality (AR) is a growing entertainment tool that has a great impact and ultimately effects the viewer in different ways. As a cinematographer you have a responsibility to set the scene and factor in issues around nausea, the atmosphere and the overall sensory experience, if done incorrectly it can negatively impact the story and consumer.

Often budget and technology limitations can creatively impact the production however the role encompasses multiple responsibilities including illustrating concept art, co-writing and even creating music. I also love the programming and software side of creation.

How would you like to see the technology for creating immersive content evolve?
We need an explosion in the industry as VR cameras advance from pro-consumer and professional easy to operate 360 cameras. The main issue is capturing spherical film with a square and flat sensor to texturise the content. In the future I’d like to see spherical sensors, however I think the force of gravity may play its part to make this physically impossible.

The greatest opportunity today is the readiness of technology and the ability to view what you’re capturing almost immediately. We have not seen the maximum of what VR can deliver, it’s is in infancy years.

My dream is consumers being completely comfortable with VR stories and completely immersed in the environment. Technology isn’t quite there yet, but the promise of 5G will mean quicker processing, rendering, streaming and sharing.

What technical challenge have you faced?
To know your craft you need to know the history, when I studied Victorian stereoscopic photography I noticed the technique we use in VR and 3D 360 today has not changed in 170 years. I replicated the Victorian finesse and built my own 3D 360 camera rig and developed specific camera software to achieve what I had been told was not possible.

Technology is constantly advancing and we have only just started discovering the potential of 360 media, it isn’t comparable to the 3D cinema wave.

The biggest technical development will be using light field cameras. I have done some experiments with holograms and with the rate technology is advancing I see this as a very exciting time to be working within the industry.

What kit do you use?
I have 20 GoPros on a customised rig made by Imaginary Pictures and camera software by VeloxIT. I’ve actually hacked the camera sensors so the final image is in a 54K resolution which I bring down to 2K and always shoot in different styles which are combined at the end.

For the project VR the Champions a large London production house quoted £300,000 but I didn’t trust what they promised to deliver as they had no proven record. In the end, I assembled a small team of savvy compositors and a pipeline director to create the VR film in a rented space in Soho. I had help from the Foundry and we worked alongside them to beta test Cara, which proved ideal.

The greatest challenge is often rendering time where you can’t see the VR video, however knowing how to read the metadata has made post-production smoother and cost effective.

How is immersive technology likely to evolve over the next five years?
Light field capture, where the camera reads light instead of colours, will change the industry phenomenally. The main issue is there will be so much data to process, but we are moving quickly.

Speedy data processing will change the way we work in VR production because once we can get lightning fast processing from computers to the cloud and back it will make the data wrangling seamless.

Lewis Smithingham

Lewis Smithingham is a creative technologist and filmmaker based in New York. He is the President of VR and Partner at 30 Ninjas and was an early adopter of VR film production. He spends his time working as a VR and 360 video film director and editor, most recently working as VR Supervisor and Editor for Doug Liman’s VR series Invisible.

What is your role as a VR 360-degree film maker?
My role has two functions within the VR department as VR Director and or Director of Photography (DOP) working closely with the VFX team because their needs often cross over with the VR departments.

I enjoy the craft of producing a story as well as the post-production components. I relish working with the VFX functions onset and the ability to build a world and tell a story.

Basically no two days are the same, I am involved in live streaming, which is great to develop the narrative and create new ways for storytelling. For example, we shot an underwater live stream for 3.5 miles in the Bahamas. It was 65 feet down on the deck of shipwreck and few weeks later livestreamed an original show with Conan O’Brien.

What skills does a VR 360 film maker require?
Because most VR gear is in beta, you need to know how to trouble shoot and problem solve at any given moment. A lot of people migrate from a DIT background, where they’re very technical by nature and managing big data is second nature.

I have had to shoot using cameras powered with car batteries, which is challenging when the director wants to do a wire or a drone shot.

Balancing technical knowledge and confidence with creativity is where the creation starts to get exciting. This medium can be complex and expensive, understanding the requirements of post-production will go a long way.

What are your essential items of kit?

• Scratch VR is a head-or-monitor based end-to-end VR toolset that allows for seamless colour grading and support for stereo and ambisonics
• Structure IO Scanner is a low poly scanner. I use it to scan places where I know the stitch is going to be messy, it’s a great head start for VFX
• Adobe Premiere with Mettle has been a lifesaver since day one
• Geometry kit with a measuring laser pointer I use this to measure stitching angles and fall-offs
• Kneepads because you will spend a lot of time hiding behind things in uncomfortable places

How is immersive technology likely to evolve over the next five years?
There are substantial changes occurring across the industry which is making the creative process fluid and a lot less expensive.

Camera technology is strides ahead of what it once was, stitching algorithms are getting better every day and the editing tools are simpler because we can be across multiple roles. Cloud stitching and in- camera preview didn’t exist a few years ago.

I can see the immediate action 5G and am extremely excited about how it will affect both capture and delivery. We will finally be able to deliver high resolutions to mobile device and it will make live shows simpler. Right now, there isn’t a good way to delivery 4K by 4K stereo to mobile devices but 5G will make this a reality.

Often something that is overlooked is how important resolution is on mobile devices, the advances in technology has created new opportunities and puts the industry in an exciting place.

What is the most challenging aspect of your job?
Every day at work is a different puzzle. The nature of VR and 360 video requires camera trickery because the camera doesn’t allow us as filmmakers to get close. Immersive content filming often means the algorithms don’t function when objects get close and factoring in stitching is always a challenge.

The biggest challenge the industry faces is that the majority of cameras don’t record a base level log, which can be frustrating and impact the colour science.

What advice would you give an aspiring VR film maker?
The best advice I could offer anyone interested is to go out shoot and learn for yourself the powers of immersive storytelling just don’t wait for it to come to you. This medium is in its infancy and there is a lot of buzz about the creative opportunities and possibility of VR and 360 videos.

David Betteridge

David Betteridge has worked as a film director across different formats from music videos, commercials and documentaries. He has honed his craft as a director, cinematographer and 360-video director for corporate clients including Microsoft, Google, Sony and Nokia. His credits include Misfits, a 360-video documentary.

What is the job function of a VR 360 video DOP?
As a director and DoP, understanding the power of the storytelling format and creating narratives to engage audiences is really important to get it right. Work has changed from simple narratives with single experiences to more exciting and complex campaigns.

How the film is going to reach the audience dictates the budget, filming style, kit used and how it will be distributed. What you need to keep in mind is that you’re not showing a story to your audience you are putting them inside it. This requires more attention to the sensory impact and crucially understanding will the viewer feel uncomfortable immersed in this environment.

What technology do you use on a daily basis?
I have an array of cameras for 360 video capturing because each job can require a different solution. Originally I started with GoPros and would build a customised rig and various lenses, which were really flexible for shooting. I also use lots of smaller sports cameras, Panasonic DSLR, Sony’s fish eye camera lens and Kandao 3D VR camera.

Surround sound recorders, different poles and grip kit and motorised dollies are fundamental.

In post-production we will have someone on set with three laptops churning through rushes and focussing on what we are doing. We then complete the fine stitching after the edit

In post-production we will have someone onset with three laptops churning through rushes and focussing on what we are doing. We then complete the fine stitching after the edit where we use Adobe Premier and autopano for panorama shots.

What is the most challenging aspect of your job?
The commissioning process can often be challenging as it is such a new format. It’s a unique process due to the technical implications, the responsibilities and check points to sign off.

The challenge here is choosing the right format and idea to remain flexible. However, immersive storytelling is a great way of sharing stories and engaging with audiences in ways not previously possible.

Three years ago the equipment was a nightmare but today it is a lot cheaper and simpler, the speed of change has surprised a lot of people in the industry. We need distribution methods to catch up and it will be interesting to see what Apple will do in the VR space. They’ve taken out patents surrounding this but have remained very quiet, the industry will grow and the result will be something more persuasive.

What technology trends do you expect to evolve over the next five years?
Stereo 360 video will be a big step for the industry as well as a lot more immersive content becoming mainstream and powerful processing tools to compliment the creation and delivery.

The ability to scan live action people into VR environments to attach the real lie action 360 video and emotion in a games environment will create a new avenue for storytelling.

As it stands, VR game engines lack emotion but combined with photogrammetry in real life 360 captured live action is literally a game changer. It will become a lot easier when the software and hardware catch up to the camera technology.

Developments in how immersive experiences can be shared will reduce the criticism on VR as an isolated experience.

Live 360 video broadcast will be very interesting but forward thinking to 10 years’ time the new forms of entertainment is unimaginable.

What advice would you give an aspiring VR film maker?
Just do it! There are so many solutions to any technical or creative issues, just get a camera, watch as much as you can and don’t be scared of messing it up.

The great thing about today is that there are no barriers to sharing your work, thanks to Facebook and native VR platforms that exist to share VR films, content can remain competitive.