DEEP-LEARNING-BASED INVERSE TONE MAPPING OPERATIONS FOR HDR IMAGE RECONSTRUCTION IN BROADCAST

Abstract

HDR is considered a major topic and will soon replace SDR as production standard. However, most legacy content is only available in lower SDR quality and must always be included into future productions. Moreover, HDR productions will not be possible in every future situation.

To adapt remaining SDR content to the future dominating HDR world, Inverse Tone Mapping is required, which creates HDR content by expanding the dynamic range of SDR material. While ordinary operations only focus on adjusting brightness values thus leading to visually unrealistic results, Deep-Learning-based approaches from the field of AI have recently led to qualitatively promising results.

The paper will give an overview of restoring dynamic range especially by learning-based and data-driven approaches of Deep Learning and will provide technical fundamentals and examples of expanded images. Furthermore, the issue of adapting these approaches to live broadcast and post-production applications will be discussed, as compliance with technical requirements and quality standards is a challenging task in this respect.

Introduction

High Dynamic Range (HDR) has been widely adopted in several markets for quite some time and is already supported by the latest displays on the consumer market. Therefore, HDR has also become more and more important in the broadcast industry and will most likely become a production standard in the near future. However, most image content from almost 100 years of television is only available in lower “Standard” Dynamic Range (SDR) quality. This legacy material, e.g. from archives of television stations will always need to be included into current and future productions. Furthermore, due to economic limitations, the application of HDR may not be affordable for every production overnight. However, not only by lack of financial means but also technical restrictions can inhibit pure HDR productions, e.g. when being forced to use lower cost cameras with small non-HDR-capable sensors.

To adapt remaining SDR and restricted HDR content or even live images to future dominating HDR content and infrastructures as well as displaying it on new HDR devices, a so-called “Inverse Tone Mapping” (ITM) or “up-mapping” is required. Since the naive use of SDR content in HDR applications leads to very bad results, ITM operations create HDR content by expanding the contrast and dynamic range of SDR material. Ordinary operations, however, only focus on adjusting brightness values leading to visually unrealistic and poor results compared to original HDR images. Especially over- and underexposed image areas with missing image information caused by technical or physical limitations of SDR camera sensors, are difficult to recover. Due to lost image information, restoring dynamic range is an ill-posed problem, which can be considered a problem of image recovery. This issue has also been researched in the field of artificial intelligence (AI) and has recently been adapted successfully to the HDR image reconstruction of still images leading to promising results. Lost brightness information can be estimated and reconstructed in a learning-based and data-driven manner using deep neural networks to mimic real HDR images. Thus, SDR material could be converted to HDR automatically, reducing cost and time.

Download the paper below