Volumetric capture: how to bridge the skills gap for broadcast professionals

Unlike traditional video, which is confined to a 2D plane, volumetric video provides the added dimensions of depth and volume, making it possible to interact with digital objects in ways that were previously not possible. Objects such as a coffee cup can be digitally captured in 3D and placed within a virtual environment almost in the same way a practical prop can be placed within an actual set. This technology has many applications for virtual reality (VR), augmented reality (AR), virtual production (VP) and mixed reality (MR).

Read more Spatial computing and immersion - a new era for volumetric production

Several technologies underpin volumetric capture, including photogrammetry, Gaussian splatting, and 4D capture. Photogrammetry involves taking multiple photographs of an object or scene from different angles and stitching them together to create a 3D model. Gaussian splatting, a more recent innovation, creates a point cloud where each point represents a small, textured circle. This method is lightweight and efficient, making it suitable for real-time applications. The cutting-edge 4D capture extends this technology to the 4^th dimension of time, making it possible to capture animated content volumetrically, providing enhanced options for animation.

Volumetric capture in production

Johnny Johnson is a Senior Creative Technologist for the StoryFutures Academy programme, teaching courses in VR, AR, VP, and volumetric capture. His career has taken him from working in traditional camera departments supporting film production and sports broadcasts through to the world of extended reality (XR), resulting in an “interesting marriage of the game engine software development skills that I had and the many years of filmmaking skills,” he says.

This background has given Johnson insight into the volumetric capture skills that will be needed on both the production and technology sides of the industry. Some of the key challenges Johnson has identified in terms of capturing volumetric content include:

• The high cost and complexity of the equipment and setups required to properly capture volumetric content currently.
• The lack of clear and accessible information around workflows for those production people interested in volumetric capture, especially smaller productions or individuals with lower budgets.
• The separate skill sets involved in volumetric capture vs manipulating or lighting the resulting digital assets. Some camera operators may not know how to handle the assets after capture.
• The time and processing required after volumetric capture to get usable assets, compared to the desire for real-time capabilities. Volumetric capture takes a significant amount of time currently, but the goal should be real-time feedback.

Johnson’s experience both in the practical application of these technologies and in teaching the techniques to other filmmakers has helped him identify the key skills needed in this production environment.

Read more IBC2024 Accelerator Project: Evolution of the Control Room

Technical proficiency in 3D modelling and animation software

Mastery of software such as Unreal Engine, Unity, and other 3D modelling tools is crucial. Johnson notes: “Unreal is the bedrock of virtual production,” enabling real-time updates and live changes on set. Understanding these tools allows professionals to create and manipulate 3D assets, integrate them into live-action footage, and ensure seamless interaction between digital and physical elements. “In terms of live updates and real-time, if you’re a filmmaker working on set, you’re expecting immediate changes in your set, your acting, your camera, or lighting,” he says. “That has to extend to the digital tools if it’s going to be suitable for films.”

Coding and scripting

While not all broadcast professionals need to be expert coders, “young people interested in digital tools, AI, and new technologies for training and production need to think about coding,” says Johnson.

However, he points out that thanks to AI, the nature of coding is changing dramatically. “Jensen Huang, the CEO of Nvidia, said that in the future, anyone will be able to code because they won’t be programming,” he says. “You’ll be having a conversation with an AI system that will convert your requirements into code behind the scenes.”

Ultimately, Johnson believes that having a strong grasp of the capabilities of the technologies available is key. “An understanding of how to get the best out of those tools is the most important skill,” he says. “Programming is quite an umbrella term, but some people would call it ‘prompt engineering.’ People need to be prompt engineers to get the most out of these things.”

Real-time collaboration and communication

Volumetric production often involves large, multidisciplinary teams. Effective collaboration and communication skills are vital for coordinating between departments, such as VFX, animation, and production. Understanding the workflows and needs of different team members ensures that the project runs smoothly and that everyone is aligned with the creative vision. Johnson underscores the importance of this, stating: “It’s about making sure the whole team can work together seamlessly and that everyone knows what’s happening at every stage.”

 

Given the experimental nature of volumetric production, creative problem-solving is essential. Professionals need to be able to think outside the box, troubleshoot issues on the fly, and adapt to new challenges as they arise. This skill is particularly important in a fast-paced production environment where technical issues can quickly escalate if not addressed promptly. “Creative problem-solving is key,” says Johnson. “You need to be able to adapt quickly and find innovative solutions to the challenges that come up.”

Education and training

To bridge the skills gap in volumetric production, there is a growing need for specialised education and training programs. Traditional film and broadcast education programs are beginning to incorporate elements of digital production, but more focused training is required. Discussing his experience teaching virtual production, Johnson explains: “We try to encourage students to look into it if that’s the area they want to explore rather than really force it upon them.”

 

Read more Webinar report: The Collision of Gaming and TV

Johnson also highlights the need for more courses tailored to the intersection of creative and technical skills: “There are very few training schemes that cater to creators with a background in computer science. People who want to get into creative industries don’t envision themselves coding their way to get there right now. They are just not aligned together at all,” he says. This gap indicates a broader need for educational institutions to develop programs that integrate coding and creative skills to prepare the next generation of broadcast professionals for the demands of production using volumetric capture and other XR technologies. “As more people see the value in these skills, we’ll likely see more courses emerging to meet the demand,” Johnson suggests. He is optimistic that as the industry evolves, educational institutions will respond by developing more specialized programs that bridge the gap between creative and technical skills.

Johnson believes that training courses are needed help address the demand for coding skills in digital production. Three important areas he has identified are:

• Providing more accessible avenues for digital production assistants and technical people in filmmaking to learn coding skills as the demand grows with the expansion of digital production.
• Offering courses tailored towards creatives with backgrounds in areas like filmmaking who want to gain coding skills to work in digital production.
• Teaching specific skills like using Unreal Engine for virtual production lighting and scene pre-visualization, which Johnson notes some established DPs are already seeking out.

And it isn’t just technicians who need training in this area. Johnson feels that production staff could benefit by developing a familiarity with the following tools:

• AI tools and systems for prompt engineering
• Tools for volumetric capture like photogrammetry
• Unreal Engine for virtual production
• Tools for 3D manipulation and lighting like Blender

Future of volumetric capture

Looking ahead, volumetric capture is set to become an integral part of the media landscape. As the technology matures and becomes more accessible, applications will expand beyond high-end productions to smaller studios and even independent creators.

Johnson is optimistic about the future: “With every new technology, there is a new opportunity to learn a new skill and create and tell a story in a whole new way,” he says. The real challenge lies in democratising access to these tools and ensuring that broadcast professionals are equipped with the skills they need to leverage them effectively. This involves not only providing training and resources but also fostering a culture of continuous learning and innovation.

For broadcast professionals, mastering the skills required for this new technology is essential to stay relevant in an increasingly digital world. By embracing the technical and creative challenges of volumetric capture, professionals can unlock new opportunities for storytelling and audience engagement. As Johnson puts it: “The ability to do more with less and tell stories in new and innovative ways is the most exciting part of this technological evolution.”

Read more Spaceport: Three-dimensional viewing experience with Freedom