No comments

ABSTRACT

Virtual Reality (VR) has recently become a hot topic in the media industry. VR systems enable the user to navigate in real or virtual worlds.

The dimensions of these worlds may range from 180 degree cylinders up to complete 360 degree spheres.

Such systems require cameras, which are able to capture these worlds and transmission systems, which enable users to view these worlds on their VR devices.

Fraunhofer HHI has developed an interactive streaming system, which provides highest quality and does not limit the image resolution neither on the production nor on the end device side.

It consists of one or several omnidirectional cameras, which can be placed into a scene like sport stadiums, theatres, fairs etc.

In HHI’s OmniCam360 there are 10 micro HD cameras, delivering a 360 degree panorama of 10.000 x 1.920 pixels.

As the data rate of such video is by far too high to be transmitted to an end device, three solutions to stream the content to end devices have been developed.

A user can then interactively navigate in the panorama.

INTRODUCTION

In recent years, patterns of media consumption have been changing rapidly.

Video material is now viewed on screens ranging in size from an IMAX cinema, through to large domestic projection and flat-panel displays, down to tablet PCs and mobile phones.

At the same time the resolution of displays is constantly increasing, 4k displays are already state of the art and can be bought at affordable prizes. Even first 8k devices appear on the market and this trend will surely continue.

Another significant change in media consumption habits is the level of interactivity that consumers are increasingly expecting.

With web-based media it is commonplace to scroll to parts of a web page that are of particular interest, or to use Google Earth to examine a particular part of the world in detail.

In a ‘first person shooter’ computer game, the player can look around in all directions, and expects the soundscape to rotate to match his viewpoint.

This level of control has not been possible with traditional video-based media, where the program director has generally determined the view of the scene with which the user is presented.

These general trends are the cradle of the recent hype in Virtual Reality (VR), which has been triggered by the acquisition of Oculus Rift by Facebook and by new VR devices like Gear VR from Samsung and others from Microsoft, Sony, Razer etc.

Industry analyst firm CCS Insight has just published a report – Augmented and Virtual Reality Device Forecast, 2015-2019 – stating the amount of AR and VR devices sold will rise from 2.5 million this year, to 24 million in 2018 (1). It forecasts that the market will be worth more than $4 billion.

VR systems enable the user to navigate in real or virtual worlds. The dimensions of these worlds may range from 180 degree cylinders up to complete 360 degree spheres.

On the production side a number of new 180 - 360 degree cameras have already been launched or have been announced.

They either combine a number of cameras to scan panoramas or the complete 360 degree surrounding or they use single cameras with wide angle lenses (e.g. fisheye) or curved mirrors (e.g. parabolic front mirror).

Single cameras are easy to handle, however their resolution is limited which results in rather poor image quality.

If more than one camera is used, a number of technical issues occur: The cameras have to be synchronized, parallax errors may occur, different sensitivities of the cameras have to be compensated and the images of the single cameras have to be stitched together.

In addition, the resulting video format may become very large, which results in problems for viewing, storage and transmission.

Fraunhofer HHI has developed a complete interactive streaming system, which overcomes all the above mentioned problems.

It consists of one or several omnidirectional cameras (OmniCam360), which can be put into a scene like sport stadiums, theatres, fairs etc.

An OnmiCam360 consists of a number for Micro HD cameras which cover a desired viewing angle.

In HHI’s OmniCam360 there are 10 micro HD cameras, delivering a 360 degree panorama of about 10.000 x 1.920 pixels.

As each camera delivers an HD frame in portrait format, the frames of all cameras have to be combined.

This processing is done by a so called Real Time Stitching Engine (RTSE), which can also render a number of HD frames from arbitrary parts of the panorama with arbitrary zoom factors, covering either the complete panorama or parts of it.

After stitching, the panorama can be delivered to fixed or mobile devices.

As the data rate of a 10.000 x 1.920 video is by far too high to be transmitted to an end device, there mainly two ways to solve this problem:

1) The complete panorama is sent to the edge of the network and the end device sends the coordinates and the zoom factor of the selected sector directly to the edge server, where this sector is rendered to a HD video.

2) The panorama is split into a number of tiles of different resolutions and the client request the tiles of the selected sector from the server. Both solutions will be explained in this paper.