No comments

ABSTRACT

Up until now, TV has been a one-to-many proposition apart from a few exceptions. The TV Stations produced and packaged their shows and the consumers had to tune in at a specific time to watch their favourite show.

However, new technologies are changing the way we watch and produce television programs. For example, viewers often use second screen applications and are engaged in lively discussions via social media channels while watching TV.

Nevertheless, immediate live interaction with broadcast media is still not possible.

In this paper, the latest results of the European funded project ACTION-TV, which is developing novel forms of user interaction based on advanced Computer Vision and Mixed-Reality technologies, are presented.

The aim of this research project is to let viewers actively participate in pre-produced live-action television shows. This expands the horizon of the interactive television concept towards engaging television shows.

The paper explains the concept, challenges and solutions resulting in a first prototype real-time demonstrator for novel interactive TV services.

INTRODUCTION

ACTION-TV [1] proposes an innovative mode of user interaction for broadcasting to relax the rigid and passive nature of present broadcasting ecosystems.

It has two key aims: (i) a group of users can take part in TV shows providing a sense of immersion into the show and seamless natural engagement with the content; (ii) users are encouraged to use TV shows as a means of social engagement while keeping themselves and their talents more visible across social circles. These aims will be achieved by developing an advanced digital media access and delivery platform that enables augmenting traditional audio-visual broadcasts with novel interactivity elements.

In Figure 1, the ACTION-TV media cloud is depicted, where different social groups can actively participate in a TV show.

By focusing on a game show use case, home viewers can take part in a quiz duel and a dancing competition, while their performance is captured and mixed with a live-action broadcast stream and distributed to their social community. This is achieved by 3D capture of the home viewer and insertion of visual appealing 3D model into the scene as a “virtual candidate”. An artistic sketch of the concept is depicted in Figure 2.

Figure 1 actiontv

Figure 1 ACTION-TV Media Cloud

The ability to interact with a program has been used to engage viewers and increase interest (and revenues) in television shows. The easiest form of interaction is a telephone conversation. Thanks to the internet, it’s now easy for people to not only be heard but also appear on television (skype, streaming etc.).

Furthermore, state-of-the-art virtual reality technologies have been reported in user engagement in television shows. In particular, HUMANOID [2] and ReV-TV [3] projects proposed to use avatar animation technology to bring in members of the audience into TV shows.

The collaborative research project REVERIE [4] developed technologies for personalizing the avatar to resemble the actual participant’s appearance through photorealistic human modelling. However, these technologies have been developed for live television shows where an actual two way conversation is possible. The possibility of interacting with a pre-recorded program is still very limited. ACTION-TV addresses this issue.

The broadcast industry is moving from an SDI workflow towards an IP based workflow. ACTION-TV capitalizes on the IPTV workflow and has developed technologies which will change the way the users can interact with television programs.

1. 3D Capture and real-time rendering of the “home candidate”

2. Ability to realistically merge a “home candidate” in a pre-recorded television program

3. Ability for a “home candidate” to interact in real time (answer questions, dance) and be a part of a pre-recorded television show.

4. Ability for a social group to watch “their” home candidate in the program and communicate amongst one another via text or video.

ACTION-TV allows a home viewer to participate as a 3D avatar in a television studio. The interactive viewer moves from a passive state to an active one. This form of interaction is new and not yet seen in current television programming.

The pre-recorded content is no longer tied to a specific time slot. A specific group can see the “transmission” at a time of their choice. The television experience becomes a big personalized game. Overall, this new concept leads to several technological challenges such as real-time 3D video processing, mixing various video streams and cloud-based computing.

In the next sections, the ACTION-TV demonstrator, the system architecture, the content production workflow, the user model generation, real-time transmission over media clouds and user evaluation results for quality of experience based on a prototype demonstrator are presented.