A SIMULATION ENVIRONMENT FOR DRONE CINEMATOGRAPHY

Abstract

Simulations of drone camera platforms based on actual environments have been identified as being useful for shot planning, training and rehearsal for both single and multiple drone operations. This is particularly relevant for live events, where there is only one opportunity to get it right on the day.

Watch Video presentation of the paper

In this context, we present a workflow for the simulation of drone operations exploiting realistic background environments constructed within Unreal Engine 4 (UE4).

Methods for environmental image capture, 3D reconstruction (photogrammetry) and the creation of foreground assets are presented along with a flexible and user-friendly simulation interface. Given the geographical location of the selected area and the camera parameters employed, the scanning strategy and its associated flight parameters are first determined for image capture. Source imagery can be extracted from virtual globe software or obtained through aerial photography of the scene (e.g. using drones).

The latter case is clearly more time consuming but can provide enhanced detail, particularly where coverage of virtual globe software is limited. The captured images are then used to generate 3D background environment models employing photogrammetry software. The reconstructed 3D models are then imported into the simulation interface as background environment assets together with appropriate foreground object models as a basis for shot planning and rehearsal. The tool supports both free-flight and parameterisable standard shot types along with programmable scenarios associated with foreground assets and event dynamics. It also supports the exporting of flight plans. Camera shots can also be designed to provide suitable coverage of any landmarks which need to appear in-shot.

This simulation tool will contribute to enhanced productivity, improved safety (awareness and mitigations for crowds and buildings), improved confidence of operators and directors and ultimately enhanced quality of viewer experience.

Introduction

Recently, unmanned aerial vehicles (UAVs) have been frequently employed as camera platforms in film and broadcast production to capture content offering an enhanced viewing experience. Drones offer flexible camera positioning with multiple angles and uninterrupted coverage, which are important for coverage of live events such as sports. In these cases, rehearsal opportunities are often limited and there is often only one opportunity for the directors, drone pilots and camera operators to conduct fly and shoot operations. A reliable and realistic simulation tool, supporting the integration of programmable foreground assets into realistic background environments, would therefore be of significant utility for planning, rehearsing and evaluating single and multiple drone operation in preparation for these types of event.

There already exist commercial and royalty-free software packages capable of flight simulation within realistic (1) or virtual environments (2, 3). Examples include DJI Flight Simulator (4), Google Earth Studio (5), AirSim (6) and Microsoft Flight Simulator (7). DJI Flight Simulator focuses on training the flight skills of drone pilots, which offers a selection of virtual background scenarios and can simulate various weather conditions. However all its environmental models are not realistic. Google Earth Studio is a web-based animation tool based on Google Earth’s satellite and 3D imagery, which can generate videos with an intuitive UI and features such as keyframe-based animation. Similar to Google Earth (8) and Microsoft Bing Maps (9), it provides sufficient resources for multiple view footage without providing any flight training or planning features. AirSim (Aerial Informatics and Robotics Simulation) is a plug-in package for Unreal Engine 4 (10) and Unity (11). It has often been used as a platform for AI and control system research related to autonomous vehicles (12). However it is not furnished with shot type grammars and does not provide realistic environment assets. Microsoft Flight Simulator 2020 is currently under development for Windows 10 and Xbox One platforms, and is claimed to simulate the entire Earth using textures and topographical data from Bing Maps. It is designed primarily for flight training and simulation rather than drone cinematography, lacking features for shot type grammar and flexible foreground object integration.

In this context, based on the preliminary work reported in (13), a new workflow for developing a fully functional CGI-based simulation tool with realistic background environments is presented here. This also includes recommended parameters for the capture of 2D environmental images as a basis for 3D reconstruction. The prototype simulation interface supports flexible planning and training with example background and foreground 3D assets.

The rest of this paper is organised as follows. The proposed workflow for building a simulation tool with realistic background environments is firstly presented, and its primary steps, environmental image capture, 3D model reconstruction and simulation interface are further described in detail together with demonstration results and images. Finally, the conclusion is outlined alongside future research directions.

Download the paper below