NAB 2018: “Incredible interest” shown in AV1

It was in September 2015, in direct and angry response to HEVC Advance and its apparent licence/royalty fee structure, that the Alliance For Open Media (AOM) formed and immediately promised a next generation codec within two years.

It was to be based on Cisco, Google and Mozilla technologies.

On March 28, strategically smart with NAB in mind, the AOM kept its promise with the launch of AV1, essentially an open-source, royalty-free alternative technology for delivering video content over the Internet.

To get a sense of the huge step in quality that AV1 offers so many areas of consumer electronics and publishing, and how the new codec was received at NAB, IBC365 talked to Gabe Frost, Chairman of The AOM, Nathan Egge, Senior Engineer at Mozilla, and Kieron Farr, VP of Marketing at Bitmovin.

Bitmovin recently took in £30m of seed funding and its three strengths are video player, encoding, and analytics.

It was the first to show an AV1 live stream, and from IBC17 it reduced the number of cores to do that from 200, and then further reduced them to 32 to enable 1080P 30fps feeds.

In November, it partnered with Mozilla to create a proof of concept full workflow, from cloud encoding to browser decoding in Firefox. In addition to its web browser, Mozilla does a ton of standards work in support of its core mission – an open acceptable Internet available to all. It pushed the royalty-free audio format Opus to standardisation from 2009, and ported quite a chunk of technology over to AV1 as a founding member of the AOM.

Frost represents 21 general members and a mighty potent governing group that consists of Amazon, Apple, ARM, Cisco, Facebook, Google, IBM, Intel, Microsoft, Mozilla, Netflix and Nvidia.

“When we look at what AV1 can offer, it is a massive improvement over H.264” - Kieron Farr, Bitmovin

Outreach and engagement
Asked if big changes in the consumer market since 2015 had confirmed AV1 as the right gambit, Frost said: “The most revealing thing has been no real change to the uncertainty in the advanced video encoding market. When we launched there was a new patent pool forming around HEVC Advanced, and since then another pool has formed.

“We have focussed on the fundamental needs of products across a wide spectrum; everything from encoders in private and public clouds to devices like Xbox, browsers, and running on mobile devices or on a PC,” he added.

This focus on the end-to-end challenge of streaming video is just one thing. “Another interesting trend that came up since 2015 is the emergence of new experiences organised around video, like mixed reality,” said Frost. “We saw more latency and live streaming scenarios given rise by open frameworks like WebRTC, and we have had a tremendous amount of outreach and engagement from companies across the spectrum who find benefits from leveraging open tool sets. AV1 is the leading option for people to deliver on those experiences.”

Step function improvement
Bitmovin quoted efficiency improvements of 35% from its trials with Mozilla, and on day one of NAB it reported that iflix had selected Bitmovin as its encoding and player partner.

“What we are excited about is the widespread adoption of video in general, and the trend coming with that involves emerging markets,” said Farr.

“In these geographies you have environments where perhaps the highest bandwidth connection is 100 Kb/sec, so what we are seeing is that AV1 allows our customers to offer higher quality experiences.

“I am not talking about sending HD video over that level of connection but we are talking about a step function of improvement in quality where finally we can access the rest of the world,” he added.

“That is very exciting and opens up new doors, and it also provides a step function increase in quality in mature markets. The ability to offer reliable non-buffering – or very infrequent buffering – for HD video on all mobile devices is very compelling.”

Bitmovin also watches the across Asia trend of rolling out super high capacity wireless networks.

Farr said: “Set-top boxes are now being manufactured with the wireless chip embedded. This includes everything you need to consume content without even having to hook back up to your Internet in your house. All these different applications are either made possible by AV1 or are significantly improved by this codec.”

High quality video in the browser
Nathan Egge explained that Mozilla had taken five or six experiments it had done into AV1. These include a new intra-prediction mode, video compression, and a constrained directional enhancement filter, which was a joint collaboration with Cisco.

“We had a couple of more radical things we brought over, that were designed as royalty-free from the outset, but they did not make it into AV1. They went through the process and showed a lot of promise: they allowed us to take advantage of advances in computational budget over previous generations of codec,” said Egge. “For example the Multiple Isometric codec (from Opus) lets us, instead of coding binary symbols, encode symbols. We can code alphabets of up to 16 symbols.”

This shows benefits in hardware decoders, because if running at the same frequency you can actually clock with more information per cycle of the hardware. The collaborative open source development aspect of AV1 is what draws Mozilla culturally.

“For the open Web what this means to Mozilla is that we can now have a high quality video standard in the browser that everybody can rely on to deliver content and services. And for our customers they get a better experience and reduced bandwidth costs,” said Egge. “It means people can innovate without having to ask for permission, which is a big thing at Mozilla. We care a lot about allowing people to create things without having to ask anybody, so AV1 definitely aligns with our mission.”

Underwriting the risks
One of the core struts of AV1 is the patent review process and legal defence fund. Frost explained that the AOM had simply adopted the W3C (World Wide Web Consortium) framework for royalty—free patent positioning.

He said. “In addition to that we focussed on leveraging the intellectual property of our member companies. We have built up within the alliance a group of world experts on both the legal and engineering sides. So we really focus on making sure that we are the intellectual property and the inventions from member companies, and we have a rigorous process to ensure that no technology gets accepted into AV1 unless it goes through a great deal of analysis.

“Because we believe that we have a codec that is built upon the IP of all our member companies, I will add that many of the member companies’ intellectual property is licensed for use with HEVC or H.264, and other leading codecs. Because they license to AV1 on a royalty-free basis we have a great deal of technology to leverage as well as the technology that was invented as part of doing the work for AV1,” Frost added.

“We believe so strongly in the work done we are willing to put the considerable resources to bear from especially founding member companies to back the legal defence fund.”

This can be seen as underwriting the risks of some of the smaller member companies, and creating a more level playing field, because the likes of Google and Microsoft can defend themselves very adequately.

Increased efficiency
Asked about the greater potential for exposing HDR to consumers, Egge said: “One of the requirements for displaying your HDR content is having the displays that do the 10-bit and the high dynamic range, and we are seeing those on a lot of consumer devices.

“We don’t see that very much on desktops, but you will soon,” he added. “Part of our delivery mechanism as we design the browser and the mobile experiences is to make sure we can take advantage of those features and push them out. Once you have a royalty-free path for getting HDR content to consumers, you will see more people providing it. Netflix is all HDR content!”

Farr wanted to raise a key perspective. He said: “Although we have those relatively new formats like VP9 and HEVC if we look across at the Web the vast majority of content is encoded in H.264. So when we are looking at what AV1 can offer, it is a massive improvement over H.264.

“As a practical matter, I am super excited that the quality and speed improvements for our customers’ users will be very noticeable. This is another rung on the ladder of quality,” he added.

Frost defined the superiority over H.264. “From a baseline perspective you are looking at 65% or better compression efficiency. That’s incredible, and when you think you can get an end-to-end experience where you can achieve better quality at all resolutions, and see the benefits of the new screens armed with HDR and wide colour gamut – that’s the real impact.”

Significant engineering resources
Bitmovin and Mozilla’s shared demo stands as a great reference point for those waiting on AV1.

“One of the nice things about that partnership is that we are able to demonstrate early on to many somewhat sceptical people that this is not just an idea. It is real technology you can use today, that delivers the real-time decoding of 1080p content right in the browser,” said Egge.

“You can use it and deploy it with a software decoder and a consumer grade laptop, and it will decode and play fine. Using Web technology gives you access to the massive market of consumers, which is a huge statement,” he added.

Farr had witnessed an incredible interest in AV1 at NAB.

He said: “I was blown away by how many people came by and asked for the AV1 demo. We have been able to show the full workflow, and the question we are getting from people is not if but when they are going to need to implement this into their workflows. The customers are sold because they have seen it working.

“The one thing I was excited to see at NAB was the people looking to experiment with hardware-based acceleration for the encoding/decoding,” he added. “It is exciting that people are investing significant engineering resources right now to bring AV1 to market very quickly.”

“We are consistently seeing an across the board 30-40% increase in compression efficiency.” - Nathan Egge, Mozilla

Not the Alliance for Open Video
Frost had seen AV1 on booths he would not have expected to show it, but of course all the AOM members companies were hammering away with the codec. Some were offering ‘different shaded’ commercial solutions in public cloud.

“It was quite incredible to see that they were going to that extent to differentiate their offerings, and it just shows the level of innovation available across the video ecosystem,” he said.

Frost had also seen tiny fringe exhibits with AV1 posters, but having had such an enormous problem to resolve – and doing it so brilliantly – does AOM move onto ‘next’ projects or stick for now? Apple is already fancying AV2.

“There is a two-part answer to that,” said Frost. “First and foremost, the alliance as a group is making sure that we are a service to create a space for our member companies in the eco system. So areas where we can create clarity and can continue to optimise the existing encoder and decoder that we have ensures that we are providing the right tools for people to build commercial implementations using AV1. It is a priority that AV1 lands very well, and the eco system has all the tools it needs.

“The second point is that a great deal of scientific work has happened in the time that we developed AV1. That doesn’t stop,” he continued.

At the launch of AV1, a set of tools that had gone through more rigorous proving work had to be chosen. Other great tools will not be wasted though.

“The development of those exciting tools continues and we absolutely expect that group to find its way into AV2, or whatever we call it in the future. But we are not announcing any second new codec at this point,” said Frost.

“We are the Alliance for Open Media not The Alliance for Open Video, and we have had a great deal of engagement outside and beyond video with people who have additional challenges,” he added. “We have a really great open source audio codec in Opus, so there isn’t an immediate need in the audio space, but it is interesting to note that the ration of bits that are used between video and audio is collapsing with the advances happening with AV1.”

Opening up new markets
Given a last word Egge highlighted Mozilla’s work on experimental encoding algorithms and a bunch of technical hurdles it can help lower by giving people better, faster technology. AV1 is a research-based codec built around proving out a set of tools, and while AV2 might take shape and address more things around Web RTC, Egge is focussed.

He said: “There is definitely more science happening, and there are things we can do to make it compelling to ship AV1.”

Farr added: “It boils my mind, the fact that we have got everything together with a codec that offers hugely improved performance. The browser companies are ready to go, and companies like us are ready to go on the encoding side: this is true improvement for the end user and it opens up new markets. It is time to motor.”

Since AV1 became a reality, HEVC Advance has decided to no longer license or seek royalty fees for non-physical HEVC content distribution, meaning Internet streaming, cable, free to air broadcast, and satellite distribution. What a difference three years can make.