Introduction to SUPERNOVA: A deep learning-based image/video quality enhancement platform

Abstract

Recently, various types of media services have drawn much attention with technical advances in the media processing arena. Numbers of IPTV/OTT based media services are becoming available through the Internet.

Watch Video presentation of the Technical Paper

This also becomes possible due to a stable installation of 5-G/LTE/3-G mobile network in addition to broadband networks. Thus, it is noted that the accessibility to media content increases and the demand for consuming high-quality media content also increases. Unfortunately, however, there still exists a lot of low quality media content that needs to be enhanced.

In this paper, we introduce a solution called SUPERNOVA that consists of deep learning-based methods to drastically enhance the quality of this media. Media content can be delivered to the SUPERNOVA platform through an API or more than one method can be selectively implemented in current local machines with GPUs. The current SUPERNOVA platform contains up-scaling (a.k.a superresolution), HFR (High Frame Rate) and retargeting functions. It is noticed that both objective and subjective performance is clearly enhanced after applying each method in SUPERNOVA.

Introduction

With the rapid increase in the demand for image/video-based media services, quality of media content is becoming a more important topic. As is well known, image/video quality degradation is mainly due to the quantization during lossy coding process. This degradation becomes especially worse as customers are located where the transmission bandwidth becomes narrower because the bitrate for the encoded media contents’ bitstream becomes lower in this environment. Another degradation case is when the spatial resolution for the delivered image/video is too small for customers to watch with their FHD or 4-K display. When this resolution degradation occurs due to instantaneous bandwidth constraints, the image/video will soon regain its original resolution, but the resolution degradation continues if the whole content in a CDN (Contents Delivery Network) or H/E Server are only stored with low resolution or low bitrate.

Until the early 2000s, most CPs (Contents Providers) produced their video contents with SD (720x480) resolution but resolutions of 4-K (3840x2160) beyond FHD (1920x1080) are currently supported in many mobile devices as well as TVs. In this case, viewers are exposed to fundamental deterioration in visual quality. Especially, the aspect ratios between SD and FHD/4K are 4:3 and 16:9, respectively. Thus, it requires more consideration when applying a linear up-scaling method such as super-resolution in order to maintain the shape of the original content. For FHD to 4-K UHD up-scaling case, 60fps is required when rendering 4-K content but most FHD content is only 30fps. In this case, methods that expand the frame rate need to be considered.

Download the paper below