MSG Sphere: Not only in Vegas

MSG Entertainment owns iconic venues including New York’s Madison Square Garden, Radio City Music Hall and The Chicago Theatre but with the MSG Sphere it aims to rewrite what it means to go to a live event. It has invented a whole new video format and production workflow for the world’s highest resolution and largest ever curved screen.

When it opens a year from now at The Venetian, Las Vegas, the $1.9 billion Sphere will not only be the largest spherical structure in the world but will claim the world’s highest resolution LED screen, sporting more than 170 million pixels and a resolution of 16K x 16K. That screen will be inside formed as a 160,000 square foot plane wrapping up and around the audience, while the exterior of the Sphere is clad entirely in another display, both of which are programmable.

There are plans to build more Spheres with one at Stratford in London announced, pending planning permission.

“Immersive has become a buzzword but I don’t think anyone has done it with to this scale or level of detail,” said Alex Luthwaite, VP Show Systems Technology, MSG Entertainment. “This will be the first large-scale venue to use multi-sensory storytelling for fully immersive experiences. The aim is to transport the audience to someplace else.”

The MSG Sphere will have 17,500 permanent seats and room for 2,500 retractable seats to bring capacity to 20,000. It’s intended to be multi-purpose, hosting everything from concerts to esports as well as experiential content enhanced with spatial audio and haptics. That means that all of the bells and whistles it has installed and the content creation technology it has developed will be put at the disposal of artists to make of what they will.

“An ambition of this scale and this ‘future forward’ raises different challenges but we have solved them all,” said Luthwaite, a Brit who has moved to LA to oversee the Sphere’s completion. “That really is the complexity and beauty of it. We’ve managed to overcome the challenge with tech solutions.

“That’s why we want to own end to end the entire process and have control of it. Because it is so bespoke we are trying to change how people perceive media development and people’s perception of going to a venue.”

MSG Sphere: 16K Screen and content

The fact that the screen is not your regular 16x9 or 4x3 video format presented the major hurdle when designing everything from content to managing colour fidelity when viewed from any angle.

“Flat, rectangular forms of traditional motion picture imaging are effectively the same at home on a 32” TV screen as they are on a 100’ movie screen, just bigger and hopefully sharper,” said Andrew Shulkind, SVP Capture & Innovation at MSG Entertainment. “In the Sphere, however, the scale differences are immense. You will never see the entire screen no matter where you sit.

“Even in IMAX you’re still aware of there being a screen in front of you. When the screen covers your entire periphery vision your brain does funny things.”

Read more Webinar: 8K content — dead or alive?

For example, producers have to be aware that removal of a horizon line in content shot in space or underwater could make people feel sick with the disorientation.

The scale of the canvas strains conventional filmmaking grammar. “The wide shape of the immersive media plane means that framing here is very different,” he said. “The close-up and the over the shoulder shot really don’t work in this format.

“One critical thing we’ve been exploring is the relation of the lens we use in the field to the field of view in the screen that audiences see. Field of view is more pertinent than focal length. We have been doing a lot of experimentation to see what works best to achieve specific psychological effects. In some cases, the best field of view for wide landscapes is a 165˚ lens, but some people and faces are better at 120˚. It is a completely new format, we are still playing with it.

“With a live music show in the Sphere, for example, we have to consider what happens to the performer on stage. How can we best feature a 5-6ft tall performer in a venue over 200ft tall, and how do we give the images meaning?”

The screen is higher resolution than any single camera can capture. MSG has built custom camera arrays and is applying techniques like tiling, uprezzing and stitching to deliver 16K resolution to the canvas.

Cameras including 8K capable REDs, Blackmagic’s URSA Mini Pro 12K are combined, after considerable R&D, into arrays for different shots.

Meanwhile, hyper-lapse (moving time-lapse) arrays required a custom built motion control rig to shoot repeatable frame accurate moves.

All of this kit is taken to location (a forest, a race track, a canyon) and mounted on technocranes or helicopters, strapped to an arm on a boat or on a Six Flags rollercoaster meaning a key design consideration is to keep it as lightweight as possible.

The exterior screen will have the capability of projecting imagery from a preset program or a live broadcast from an event going on inside the facility.

Underwater scenes for one video were shot using 15 prosumer cameras. “You can’t get too close to subjects,” said Shulkind. “Even small details in the water can make stitching challenging.

“The 180-degree field of view and resolution is so unforgiving the final image has to be seamless. There’s no tolerance for artefacts or faulty stitching.”

MSG Sphere: Into Space

Initially, Sphere attractions will be less CG and more grounded in what feels natural – epic landscapes, for example. It has signed with NASA to send camera equipment to and from the ISS on three separate launch missions. Astronauts will conduct onboard testing.

The intent is to use the microgravity conditions on the ISS and the harsh environment of space to develop a custom system capable of capturing the Earth “at a level of detail never before possible, and that can potentially be used deeper in space.”

Back on earth, boxing, basketball and esports could be held inside the Sphere. MSG owns the New York Knicks (NBA) and the New York Rangers (NHL), as well as esports organization Counter Logic Gaming.

“We are working out how take the concept of watching esports players and flipping it around and having the audience play,” said Luthwaite.

MSG Sphere: SDI-free network

The whole video network at Sphere is driven by ST 2110. There’s no DisplayPort and no SDI.

“The first reason is we’re streaming multi-layer 12-bit 60fps uncompressed media and the second reason is the sheer scale,” said Luthwaite. “We need to get a lot of bandwidth across a very large space. Plus, we want to incorporate live camera feeds from broadcast trucks to mix together on site, we wanted to stick with one standard.”

There is also the facility to incorporate live feeds picture in picture into the giant canvas.

“We wanted to be IP based and not run copper cables. There are so many channels of data going around. The more traditional video cables were not going to be able to transport it.”

MSG Sphere: 3D spatial audio

Developed in partnership with Berlin-based Holoplot, Sphere will house a custom spatial audio system – Sphere Immersive Sound – using 1,600 permanently installed speaker panels each composed of approx. 100 individual speakers, resulting in over 160,000 channels of audio.

These speaker panels sit behind the LED media plane, which is acoustically transparent for “concert grade” audio – a feat of audio engineering that has been extensively modelled and tested.

Sphere Immersive Sound is combining 3D audio-beamforming and wave field synthesis. The beamforming allows for precise shaping and steering of beams to create sound fields that optimally cover audience areas of any shape and size. Wave Field Synthesis enables sound localization “with lifelike distance” and directional perception of audio objects.

Read more Object-based audio – immersive experiences and personalisation

For example, an audience member could hear a whisper that sounds like someone is talking directly in their ear. Guests sitting in different sections can hear different sounds (eg: languages, instruments) – expanding the possibilities for customised audience experiences.

“A lot of the work we’re doing is creating a guide for what works and what to avoid for when artists and engineers programme this space,” said Stuart Elby, SVP Advanced Engineering at MSG Entertainment. “If someone came in with content mixed with Atmos we would be able to transcode that into Sphere but really it is the Sphere that has creative tools for audio engineers and artists to produce unique sound experiences.”

MSG Sphere: In-seat Haptics

The seats include haptic devices which allow artists to direct frequencies ranging from ultra-low rumbles to pitches of 500hz directly to individual seats, creating an additional layer of immersion beyond the audio and video.

A proprietary wind system has been devised to enhance the illusion of being in the natural world. It can be adjusted from an idle breeze “that you’d hardly notice” to gale force winds.

“We’ve quite a bit of IP around real wind effects designed to convey realism to the audience,” said Elby. The same system is used to disperse scent. “The typical length of screen content will be 45minutes to an hour, so we’re not trying to totally overload your senses like a theme park ride.”

MSG Sphere: LED screen design

If the interior is somewhat more of a traditional LED screen, the 580,000 sq ft exosphere presented new challenges, not least to be heat and UV resistant and to deliver uniform visuals across a convex surface. It’s composed of 1.4 million 80mm ‘pucks’ (or single pixel) that each contain 48 diodes.

The team faced similar challenges in terms of the pixel pitch except this time on a convex surface so that the pixels don’t appear wider at the equator than they at the poles.

Luthwaite said, “We made mega panels, trapezoid in shape, to triangulate the exterior surface, interlacing the pixels with a 225mm offset spacing. This increases your perceived resolution. The image resolve happens closer because of it.”

“Certain elements have been done on other projects but never to this scale or complexity and never all in one place. With such a large exterior, that it is even visible under flight paths, we want no single point of failure. We have to make sure there are redundant data paths all over the surface.”

Testing for content production and viewing has been taking place at MSG Sphere Studios in Burbank, which is a quarter-scale prototype of the Las Vegas Sphere.

Testing for brightness and image resolve on the outer screen involved building another small sale prototype and viewing it half a mile away.

MSG Sphere: F1 opening

The venue is timed to open just ahead of the Las Vegas Formula 1 GP which has a track that surrounds the Sphere. During the race F1 will take over the exosphere display for race-related content.

“There is a massive amount of connectivity and flexibility in the building because we want to be somewhat agnostic to the show,” said Luthwaite. “It is up to the creatives to come in and use any or all the tools at their disposal to create the best content.”

Read more 8K TV: UHD, HDR and the resolution wars