Flexibility, scalability and complexity: Inside BBC Wales’ all-IP Central Square facility

At the beginning of this year, S4C, the Welsh language broadcaster, moved its TV and online operations to the new facility. BBC Wales provides playout and other technology services to S4C.

Nearly five years in the making, construction began on the £120 million Foster + Partners designed building in 2015. The broadcasting infrastructure was designed and built in a collaboration between BBC Wales, BBC Major Projects Infrastructure, Grass Valley and db Broadcast.

Relocating 1200 staff and all the technology that supports a national broadcaster would normally be a significant undertaking, however this project had more challenges than normal – Central Square is one of the first fully IP-based facilities in the world. And if that wasn’t challenging enough, seven months after the migration into the new building began, the whole of the UK went into lockdown in response to the Covid-19 pandemic.

The broadcaster

“It’s been a strange time,” says Roger Crothers, Head of Technology for BBC Wales. “We went live just before the second wave of Covid, and there was concern that another lockdown might prevent us from moving into the building due to potentially increased staff absence, so the decision was made to accelerate our moving in date. We closed down our existing broadcast centre and S4C closed theirs and that freed up the staff we needed to fully operate out of Central Square with a lower risk of disruption.

“We have been on-air for almost two years, although understandably, not everyone has been in the building – Covid restrictions put paid to that. There have been times when we’ve only had around 50 people in the building. But now it’s beginning to ramp up as restrictions are eased, so we’re seeing 400 or 500 people now. This is much closer to the numbers that we originally anticipated, but we still haven’t achieved full occupation yet.”

The accelerated timescale had a significant impact on the training required to make the transition from traditional baseband technology to new SMPTE 2110 all-IP infrastructure underpinning the facility.

• Read more IP facilities: Lessons from the front line

“It meant that we had to reduce the time available for technical training,” he adds. “This has made things difficult as the infrastructure in Central Square requires a highly trained team to support and manage. The technology used is significantly different to what we had in our old broadcast centre, so most engineers had a steep learning curve. We had planned about three months of very intensive training immediately prior to go-live, but in the end we crunched it down into six weeks.

“2110 is the backbone of the network – all of our systems are integrated, which offers fantastic benefits but can also make things more complicated, because it means that there’s so many dependencies between systems. In our old broadcast centre, systems were more isolated and even air-gapped so problems were more contained. We have also now automated hundreds of workflows which requires an in-depth understanding of the steps involved in each in order to troubleshoot issues when they occur.”

“That complexity has made the support burden greater and can be as difficult to troubleshoot as the IP broadcast infrastructure. Our support contracts are now far more comprehensive but it’s necessary due to the software nature of many of the systems and the dependencies they have with each other. We also need the latest security patches and updates in order to ensure our systems remain well protected given that they are all linked via the IP protocol.”

“The technology used is significantly different to what we had in our old broadcast centre, so most engineers had a steep learning curve,” Roger Crothers, BBC Wales

Discussing the impact the new technology has had on production operations, Crothers says: “It became apparent during the tech design phase that moving to IP would have some impact on things operations staff would often do. It is not a trivial task to add native IP devices to the IP core as it requires an understanding of the IP infrastructure which many of these staff may not have. This has meant that there is a greater reliance on engineers to not only do the work, but to also be responsive to the needs of the productions.”

The flexibility of the 2110 infrastructure is undoubted, however, as he notes: “It’s format agnostic so we know that any video or audio we acquire can easily be routed through our infrastructure. It’s resilient - with 2022-7 we can tolerate card or fibre failures with no impact on transmission. It’s also scalable so we know that we could acquire in higher resolutions and the infrastructure can cope, subject to bandwidth limitations. We also have the benefit of media gateways with IP and SDI connections that allow us to take IP flows and pass them to SDI devices or take SDI flows and pass them to IP devices. This has ensured that we can continue to use some existing SDI technical assets such as vehicles or even new devices that do not natively support 2110 seamlessly.”

“I think the BBC should be applauded for their courage in the amount of new technology they rolled out,” Martyn Humphreys, Grass Valley

The vendors

The 2110 standard was not fully complete when the Central Square project began, therefore the project required close co-operation between the BBC and its suppliers. The overall architecture was done by Grass Valley, and the task of building everything fell to db Broadcast, a UK-based systems integrator.

Martyn Humphreys, Director, Cloud Platform at Grass Valley, reflects on his experience with the project. “We had a team with the BBC Major Projects and BBC Wales teams connected to the Grass Valley project team and db Broadcast. So often suppliers or vendors and customers go into things as partnerships, and that isn’t always born out,” he says. “But actually, in this project, I think it really was. We’re very, very proud of how much of a partnership it was. I think the BBC was a very positive partner to work with, as we worked through the teething problems that one inevitably has with those early stages of technology.”

Mike Bryan, Technology Director at dB Broadcast, notes that the flexibility of the 2110 infrastructure is already being taken advantage of. “They’ve already been able to do some UHD projects. Even when we started the project, things like that were not 100% in scope. There was an expectation that they might want to work with those sorts of formats in the future,” he explains. “We’re able to enable that on the fabric really easily and that’s part of the joy of IP. That is the one of the big benefits to BBC Wales - they’ve got a format agnostic platform and can do things which they wouldn’t have been able to achieve easily using SDI.”

“I think the BBC should be applauded for their courage in the amount of new technology they rolled out,” says Humphreys. “And how from the ground up, they said, ‘let’s look at how this should be done for the next 10 years, not how we’ve always done a broadcast facility’. It was wonderful, because we were on the cutting edge - as the standards came out and the products came out, we were implementing them and working on stuff together, which was great, great fun. Sometimes it felt like we slipped just over towards the bleeding edge side of cutting edge, but we got there in the end, and I’m really proud of what we delivered.”

“You know, I’d be lying if I said there were never any tensions,” adds Humphreys. “There are always issues that come up, but there was trust among the team. We all knew we were all pulling together for a common goal. I think that trust lasted right through to the end. It’s really nice to look back on actually.”

• Watch Embracing IP: The role of SMPTE 2110 in building flexible facilities