VR may have been something of a slow burner in terms of technology buzzwords, but its recent rebrand as spatial computing may well signal an uptick in interest for a host of reasons, discovers Mark Mayne.
The tide appears to be turning with a slew of big tech players fielding a new range of VR/spatial computing hardware this year, but will it be enough to achieve a breakthrough in a space that has seen many a false dawn? We look at some of the key innovations driving volumetric production in 2024.
Changemaker - The Apple Vision Pro
The biggest news in the space is arguably Apple’s Vision Pro, partly due to the dominance of the Cupertino tech giant in associated markets, and partly due to the galvanising effect on the competition. As has been the company’s playbook since the success of the iPhone, the Vision Pro delivers a market-leading set of specs, wrapped in an aspirational magnesium and carbon fibre design that will enable many to look beyond the considerable $3.5k price point.
The Apple Vision Pro’s MicroOLED panels offer a 4K viewing experience with 3,386 pixels per inch (ppi) (the consumer standard for 4K UHD is 3,840), which compares favourably to the current competition, such as the Meta Quest 3 (approx 1,218ppi), and the HTC Vive Pro (950 ppi). Apple has also included video pass-through to allow the Vision Pro to be used as an AR device, overlaying apps and visual cues over our everyday surroundings, as well as a traditional VR display for entertainment, business data and more - Apple CEO Tim Cook has said several times how important he believes AR will be in the future.
Simon Windsor, Co-Founder and Co-CEO of Dimension Studio agreed: “At the moment, it’s still slightly unusual to walk around with headsets or smart glasses on. But many would feel comfortable doing so at home, as tech like the Apple Vision Pro offers exciting potential to entirely reimagine how we experience film, TV or live entertainment, and we’re seeing wide ranging interest and exploration from entertainment studios and media companies, which could propel film and TV content into a whole new realm of creativity.
“Apple is making a play for how we might enjoy and consume content in the living room, saying ‘hey sit back and enjoy a much richer and awe-inspiring immersive experience.’ With the use of AR and VR applications on the Apple Vision Pro, we are being provided with an easy and convenient step towards more immersive experiences and naturally this will impact how the content we consume is made.”
Read more CES2024 Preview: VR, XR, Transparent Panels and Inevitably, AI
Apple is not alone in shaking up the market though, with Qualcomm making a big splash at CES2024 with the new Snapdragon XR2+ Gen 2 VR chipset ‘a single chip architecture that unlocks 4.3K spatial computing at 90 frames per second’. Tellingly, one of the launch quotes was from Samsung, referencing Google, implying there will be something VR-related to see from the pair soon: “Samsung is thrilled to collaborate with Qualcomm Technologies and Google in revolutionising the mobile industry once more.” Inkang Song, Vice President and Head of Technology Strategy Team at Samsung Electronics, said in a statement.
Not content with supplying Apple with the Vision Pro’s MicroOLED panels, Sony has also developed a VR/AR headset specifically designed for the creative community, in partnership with Siemens. The immersive spatial content creation system, which includes an XR head-mounted display equipped with high-quality 4K OLED Microdisplays and video see-through function, will be available later in 2024 according to the company.
The power of the ecosystem
In spite of Meta’s best efforts over the years, the ecosystem has often been one of the challenges around wider adoption of spatial computing, and a key inhibitor to achieving wider success. Apple promised more than 600 immersive apps and games for the Apple Vision Pro at launch - via the App Store platform - with launch examples including sport and entertainment.
These include an NBA app, where basketball fans can stream up to five games live or on-demand simultaneously, and PGA TOUR Vision, which uses real-time shot tracking layered on top of 3D models of real golf courses alongside key stats to bring the tour to life in a user’s space.
“With Apple Vision Pro and the power of spatial computing, Apple is teeing us up to deliver an innovative and immersive way of experiencing professional golf,” said Scott Gutterman, PGA TOUR’s Senior Vice President of Digital Operations in a statement. “PGA TOUR Vision, the first golf app developed for Apple Vision Pro, takes fans inside the ropes and directly onto the greens of the world’s most iconic courses, from Pebble Beach to TPC Sawgrass, no matter where they are.”
Entertainment wise, there’s Disney+, IMAX for 2D and 3D content, Warner Bros. Discovery’s Max with select titles available in 4K and Spatial Audio with Dolby Atmos.2, and of course Apple TV. The latter includes Apple Immersive Video, which offers 180-degree, 8K 3D recordings captured with Spatial Audio.
Windsor continued: “For filmmakers and storytellers, giving users the ability to select their screen size and virtual viewing environment unleashes limitless creative possibilities. As such, we will see creators increasingly embracing 3D and mixed reality production techniques to create 3D content that extends out of the screen enveloping viewers in full virtual reality. In short, users will step through the frame and be present in their favourite film and TV show rather than watching from the outside, or 3D content will step into homes and seamlessly understand our setting and context.”
Volumetric formats emerging
One company that is firmly on the side of the ecosystem pipeline is V-Nova, which acquired Parallaxter late 2023 to gain the company’s PresenZ, a volumetric movie format that can prerender 6DoF images from a standard animation pipeline, according to V-Nova.
The acquisition has lent additional drive to a company that already had a firm stake in VR/XR, via its LCEVC MPEG-5 video enhancement standard and VC-6 production codec - with a slew of concrete partnerships and use cases planned throughout 2024.
“At the moment that we are working with most of the majors - Dreamworks is the one that I can cite because they have already presented some short content [using the format] - they showed a conversion of a scene from How To Train Your Dragon and a couple of scenes from Abominable - those scenes were only the beginning”, Gianluca Meardi, SVP Head of V-Nova Studios told IBC365.
A key challenge for the VR industry has always been the delivery mechanism, where exceptionally large files are necessary to deliver high-resolution 360 content, making on-demand a logistical roadblock. Another bottleneck is the processing power of the end device - which as Meardi touched on, is nothing like as potent as a high-end GPU or a games console, making processing the stream in advance a crucial component of a practical delivery system.
“This is why we are enabling cloud streaming so that our content will be delivered and processed in Cloud - eg, from Nvidia cloud XR, and then using our video compression technologies like MPEG-5 LCEVC that can be streamed to the final device via WiFi at home. [We] demonstrated it at the last IBC and in that case, you do not need anything better than a MetaQuest 3 to experience these immersive movies. In the future, in my opinion, the main business channel for all our content will be cloud, like Netflix for instance or any OTT service”, explained Meardi.
He also pointed out that the recent SAG AFTRA strikes underscore a potential opportunity. “If you consider the strikes that happened some months ago, it stopped the production of new content and now a lot of big companies in Hollywood are suffering from a lack of new productions. This is a problem for them, but maybe an opportunity for us because there are animation companies and visual effects companies that can look into converting and monetising existing content in our new format…”
Real world broadcast applications emerge
While end-user traction for spatial computing might be at the beginning of a growth spurt, industry use cases are well documented in areas such as automotive design, and training in complex (and potentially hazardous) areas such as oil and gas. However, stunt and production company ATS is also bullish about the possibilities of VR, having used it to design the set for the first US Ninja World Cup.
“All of the VR models we view are built in SketchUp. We have used SketchUp for around 10 years to design the foundation of our sets, using it in every step of the process from pre-vis, to budgeting, prototyping, final design, and even the actual on set build. I discovered that another company named Sentio created a plug-in for SketchUp that allows quick and easy viewing of SketchUp models in VR, which is what started this whole process”, explained Nate Moore, CEO, The ATS Team.
“My initial exploration into VR was actually spurred on from a conversation during a pitch where the network was interested in a VR component to a show. I had little knowledge of VR so I purchased a MetaQuest 2 to learn more. I originally just viewed one of our 3D models from SketchUp for fun, but I quickly realised how helpful it could actually be. I loaded a model for an upcoming Ninja Warrior season and I noticed some mistakes that we missed when viewing the 3D model on the computer. There are certain details like the height of an obstacle or a deck that is very easy to notice in VR that you wouldn’t catch on the computer unless you used the measuring tool in the program but that is slow to do for every deck or obstacle. In VR I can quickly stand on the platform and immediately know if heights are off because the perspective I am viewing the model in is based on my actual height.”
Moore also believes that VR can enhance communication around set design projects, managing expectations and ensuring that the finished product is exactly as anticipated by the show or series producers.
“It is also hard during a pitch sometimes to communicate a world you have built and been working on for months in a matter of moments, but viewing the set in VR can quickly communicate what your vision is without any misunderstandings. The same process would also be helpful for communicating with current clients about what our plan for a set is and they would know exactly what to expect when they walk on set. The less irritating surprises producers get when walking on set the better.”
He continued to paint a picture of the future use cases for VR, one that is all the more likely as devices become more powerful, readily available and - overall - commoditised. “The technology is progressing incredibly fast. I don’t believe it will be too long before we will be able to go on scouts to actual locations and view full size 3D renders of sets on their actual locations. This means you would be able to walk around a full size set and know exactly what not only the set pieces will look like, but also what all of the backgrounds will be. That means that instead of building everything on site then moving it around the location with heavy equipment while the director or DP decide on the shot they want, that can all be done virtually ahead of time, so when our crew arrives we already know exactly where to set it to get the exact shot they want. Small innovation like that can lead to increased savings in time and money”, summarised Moore.
Overall, it seems clear that VR or spatial computing has turned an important corner in its evolution. From prospective gamechanger to buzzword, the typical technology arc has been well trodden, but with emerging consensus on real-world applications, formats and increasingly competitive device manufacturers there seems to be a clear direction of travel - immersion here we come.
1 Readers' comment