State-of-the-art of extended reality in 5G networks

No comments

Abstract

Virtual Reality (VR) and Augmented Reality (AR) both retain importance in the media industry. However, the overall market adoption of VR technologies is still slower than expected. At the same time, industry efforts are shifting towards AR. This is also foreseen as a much larger market in the future, particularly for AR on mobile devices which are more widespread compared to VR devices. On the one hand, the AR explosion is due to recent developments in mobile technologies and on the other hand this is based on the promises of 5G networks. 5G, as a wireless communication infrastructure, will be able to satisfy the resource-hungry demands (e.g., in terms of required bandwidth and low delay) of new Extended Reality (XR) services and applications. This paper will cover use cases, architectural, protocols and codec aspects for XR systems over 5G networks and their state-of-the-art in 3GPP and MPEG standardization organizations.

Introduction

5G promises new capabilities compared to existing network architectures, including higher bandwidth, lower latencies and new functions such as slices, virtualization and edge computing. Immersive media and extended realities quite often are viewed as one of the key experiences that are enabled by 5G, taking into account the requirements in terms of bit rates, latency and ubiquitous availability of such services. Immersive media and 5G is considered a combination that will enable new services and opportunities within the mobile ecosystem, both in the entertainment and gaming environments, but also for new verticals such as industrial services, public safety and automotive.

To structure the work on 5G and XR, 3GPP has launched a feasibility study to identify use cases, technologies, and possible gaps that need specifications for interoperable services. In the context of this paper, Extended Reality refers to all real-and-virtual combined environments and human-machine interactions generated by computer technology and wearables. It includes representative forms such as Augmented Reality, Mixed Reality (MR) and Virtual Reality and the areas within the continuum among them. The levels of virtuality range from partially sensory inputs to fully immersive VR. The ultimate objective for immersive media is the experience of Presence providing the feeling of being physically and spatially located in the virtual environment. The sense of presence provides significant minimum performance requirements for different technologies, such as tracking, latency, persistency, resolution and optics. Such experiences may be consumed on smartphones, HeadMounted Displays (HMDs), AR glasses, heads-up displays or new emerging form factors devices.

In terms of integrating XR into networks, different aspects need to be considered. Among others are the (cumulative) applications data download size (e.g., Fortnite has several GBs in download) or streaming immersive scenes at up to 100 Mbps. In advanced systems, the XR pose is not only processed in the device, but it is sent to the network in order to adapt to the current viewport or to a predicted viewport. In order to maintain the immersive experience, the pose needs to be processed in a matter of few milliseconds (typically 20ms). If the processing is delegated to the network, it either requires very low processing and communication latencies, or a smart separation of network-based pre-rendering and local rendering (also referred to as split rendering).

This document provides updates compared to what was presented 12 months ago and provides a summary of the status of the study in 3GPP on XR over 5G as well as related aspects.