Speaking volumes

“Our technology is about the highly realistic reconstructions of humans,” says Dr. Oliver Schreer. “We capture them with quite a complex camera system, and with the video data we create a dynamic and realistic model of the person.”

A great number of scenarios and applications have emerged from CG render engine technology, but the volumetric camera system is a big star.

“We created a prototype capture system in October of 2017, and that was a cylinder with a diameter of 5m which was equipped with 32 high res cameras,” says Schreer. “Compared to other volumetric video studios we do not use green screen as the background. [Instead] we integrate lighting in the background: a wall lit with LED panels mounted behind it provides the illumination of the person as consistent and homogenous illumination from any direction.

“There are no shadows coming from the light itself because the human is illuminated from every direction,” he adds. “In 2018, because there was tremendous feedback on our very first trial productions and our technology, we decided to create a spin-off company called Volucap. The ownership combines Arri, Interlake Systems, the famous German production house UFA, and Studio Babelsberg.”

In June 2018 a second production-capable studio was opened in the Babelsberg Filmpark.

“This has a revised version of the prototype capture system that we have in our labs. Some parts of the technology and concept are the same, but there are differences,” says Schreer. “It is much more of a professional capture system at Babelsberg because it uses the latest hardware. We have increased the dimension of the diameter of the rotunda, so it has more capture and acting space.”

The main principle of the capture system is identical, however.

“[The Volucap partners] had not done any previous work on volumetric video, simply because it relies on this complex capture system which nobody has except for us,” says Schreer. “Our very first test productions in the prototype studio were produced together with UFA, which was very interested to use volumetric assets for future VR or augmented reality applications.

• Read more: New adventures in AR

“The first test for 360° volumetric video was the production An Entire Life,” he adds. “This presents a scenario in a film studio from the 1920s; you enter the studio and become part of a production with the director and an actress. Both were captured in our volumetric video studio and integrated in this VR experience.”

At IBC, Schreer will be using state of the art VR headsets for display purposes, with a real-time render engine allowing him to demonstrate the VR app.

“For IBC last year, because you can navigate in six degrees of freedom in virtual space, we created an area of about three by three metres so people could walk around a bit,” he said. “They could explore the scene from any direction and observe both the director and actress, and how they are performing.”

Challenging capture
The capture system consumes 1.6TB per minute, which set a scientific challenge.

“The sequence of meshes for example is just about 7GB, so the research challenge is to do some clever 3D video processing and also mesh processing and mesh reduction, which keeps all the details but reduces the amount of data,” says Schreer. “The 3D Human Body Reconstruction (3DHBR) technology we created and the production workflow will now become beneficial to wider society.”

“The 3DHBR technology we created … will now become beneficial to wider society”

There are great apps outside media. “One is in the cultural heritage domain, where we will capture famous people from history ‘to keep them alive’,” says Schreer. “We are just making a production with a man who survived a concentration camp during WW2. You are in the scene and you experience this old man talking about what he suffered. This is a different perception of history.

“A second app for volumetric video is medical,” he continues. “There is a project where we capture humans during surgery, and the idea is to create photorealistic dynamic 3D models of patients on the operating table. Students and doctors can use the virtual realistic reconstruction volumetric asset, and not rely on the real patient.

“This will become an important field of application,” he continued. “It is quite amazing for us to develop something, a complete workflow, that is really useful for both society and the business domain.”

Long straight journey
Tracking back 25 years, Schreer has been working constantly in the same field. “The core technology is 3D computer vision and 3D vision video processing,” he explains. “We started 15 years ago with simple stereo camera systems. We constantly extended capture set up.

“Our first projects were done for 3D video communications, to capture a person, do the 3D analysis, and create a novel virtual view to perceive eye contact for video conferencing,” he adds. “This was just a head and shoulders set up in where we captured the person in front of a screen.”

In 2010 the camera set-up jumped to 16, and then onto 21.

“Even four years ago people were very hesitant with our technology, because we were so advanced”

“Finally, in 2017 we closed the loop and we set up the full 360-degree capture system,” says Schreer. “It was a very long path. Even four years ago people were very hesitant with our technology, because we were so advanced. We used a lot of PCs and complex hardware, and complex algorithms.

“But due to the advent of VR we recognised that there is huge potential for our technology,” he continues. “It was really kind of lucky, but also down to the long-lasting research on the same topic that brought us to the leading position in this field.

“If you develop such a complex workflow you need skilled people, and we were also lucky with this because the market for the universities of high-quality enthusiastic researchers is pretty thin,” he adds.

Recognition and beyond
For their research on 3D Human Body Reconstruction, Oliver Schreer and his colleagues Ingo Feldmann and Peter Kauff won the 2019 Joseph-von-Fraunhofer Prize.

With Volucap established, people asked Schreer, Kauff and Feldman if their development was finished.

“I always answer that we are now at the beginning of a long journey. There are so many aspects to further improving the quality of a unique system and the data which nobody else has,” says Schreer. “Additional functionalities will make it more flexible and faster in production. The nice thing is with Volucap we have feedback from real productions.

“New challenges arise because creative people who want to use volumetric video come up with ideas and scenarios that require further algorithm development,” concludes Schreer. “We are at the start of an exciting journey.”