Virtual production’s ascendancy has been so dramatic and far-reaching that it’s difficult to think of a recent parallel in either broadcast & media or pro-AV. Perhaps the closest comparison – in broadcast, at least – is 4K/UHD, which is now being used routinely across all manner of output. But even that can be seen as a logical progression in resolution from HD. By contrast, virtual production represents a major new area of the business whose presence has skyrocketed in just a couple of years.
It does not take out long to work out why. Green screen replacement and other technologies that would ultimately fall under the catch-all of ‘virtual production’ have been growing in profile for a while – comfortably before the onset of the pandemic. But no one disputes that Covid-19’s sudden necessitation of new practical limitations, especially with regard to complex external and location shoots, provided a huge catalyst.
It’s also been very beneficial to many companies whose other activities were affected by the pandemic. Richard Mead is CEO of video processing manufacturer Brompton Technology, which recently secured a £5.1m investment from Connection Capital to further grow its presence in areas including virtual production. “There is no question that the growth of LED in virtual production helped us at a time when the live event industry was fully suppressed,” he says. “It would have been a much tougher time for us without that, and we consider ourselves very fortunate. We pivoted almost overnight to focus completely on supporting the emerging in- camera visual effects market, but were helped by the fact that performance on camera had always been a key priority for us because so many major live events are televised.”
There are plenty of comparable stories across the AV world, with companies suddenly discovering that a significant new market was opening up for them. Moreover, no one expects the pre-pandemic reliance on location shooting to return to its former level; virtual production as a primary industry technique is here to stay. All of which means that with work in live events, museums and other visitor attractions – many of which are also using large-scale LED video products – picking up again, the next few years are looking very promising for those companies who have established themselves in virtual production (VP).
If Citizen Kane was the defining film of cinema’s first golden age, then there is a good case to claim that The Mandalorian holds a similar status for the dawn of VP. The result of a long process of research by Industrial Light & Magic in conjunction with video game developer Epic Games was a new VP visual effects technology, StageCraft, based on Epic’s Unreal Engine gaming system. Incorporating large LED video screens on which digital environments provide a real-time background for performers, the technique wowed audiences when the show debuted on Disney+ in late 2019. Virtually everyone who was interviewed for this article mentioned the series and the increasingly galvanising effect it had on the industry as the severity of the pandemic became more evident in the early months of 2020.
Christina Nowak is director of virtual production at Anna Valley, which is a provider of AV to the broadcast, entertainment and live event industries. “Virtual productions’ rise in popularity is derived, to a large extent, from the PR behind The Mandalorian,” she says. “The way they applied already existing virtual production techniques was both high profile and perfectly timed with the lockdown. At the time, productions needed to rethink how they operated – they needed to condense the talent and crew but get the same result – and virtual production solved a lot of these problems.”
What makes this trend more remarkable is the speed at which it has developed. Charli Harding, client & brand director at LED screen hire and technical event producer iMAG Displays, notes that when the company was first contacted about a VP project in 2019, “green screen replacement was the term being used; ‘virtual production’ was still to come.” With lockdown ruling out live events, it was therefore fortunate that “the same company [we worked with in 2019] phoned us up again and instigated a new project – since when it’s been 2.5 years of supplying a lot of VP solutions, which has been great.”
As might be expected, the approaches taken by different companies can vary quite a bit. There are those like Anna Valley who have built their own multi-purpose studios that enable them to carry out projects on-site. Then there is also the partnership approach, whereby companies look to collaborate long-term with a production house, which is the model chosen by iMAG and its film partner, Treehouse Digital.
“We decided not to go down the fixed studio route as we wanted to be flexible,” explains Harding. “Treehouse Digital has a studio two miles from us, and we are able to do a lot of work there. But we also frequently work at Shepperton Studios [in Surrey] or Sky Studios in Heathrow.”
For those who have built their own studios, one of the key challenges is to design them with enough flexibility to be able to accommodate the requirements of an area of technology that is still evolving rapidly. For instance, camera and lighting systems manufacturer ARRI recently opened a new VP facility, ARRI Stage London, which was designed by ARRI Solutions and delivered as a collaboration between ARRI and Creative Technology. The 708sqm stage showcases the latest in-camera visual effects technologies that ARRI and its partners have to offer for creative projects, but has also been designed to test, streamline and refine integrated workflows in a real-world VP environment. It is also being used for a variety of feature, episodic television and commercial productions.
“To accommodate all these different production requirements, it was of paramount importance to design and build a volume that is extremely versatile and flexible,” says Stephan Ukas-Bradley, VP solutions – Americas at ARRI. “This was accomplished by utilising movable LED wall segments, motorised suspension systems, and traditional lighting fixtures to complement the environment created by the volume and create some additional depth and dimension.”
The confidence that the TV and film industries now feel about VP is also borne out by the remarkable level of expansion taking place at many leading film studios, including well-known names like Shepperton, Pinewood and Garden Studio (for a full list of current UK projects, see https://thestudiomap.com/new-film-tv-studios-under-construction-in-the-uk/). Many will require comprehensive VP technology solutions, suggesting that vendors could be on the verge of a lucrative new era.
“The number of stages being prepared for high-end production in the UK alone is quite remarkable – you’re almost looking at a doubling of sound stage [capacity] and so many are based around virtual production,” says Paddy Taylor, head of broadcast at Mark Roberts Motion Control.
“Adoption was much more rapid and widespread than it likely would have been [without the pandemic], but there is no sense of it being a fad or a bubble,” says Mead. “The technique clearly has so many advantages, including the experience for the creative team, speed of working and, therefore, overall cost that it is now an established part of mainstream film and television production.”
Nonetheless, as a fairly new area of technology, it’s to be expected that there are still gaps in the product ecosystem that need to be addressed. Hence multiple vendors and service providers alluded to a focus on streamlining aspects of the process and improving overall integration.
Ukas-Bradley notes: “A key focus for ARRI is to ensure smooth, dynamic integration at all stages of virtual production workflows. For example, when it comes to real-time metadata transfer, ARRI has developed a lens metadata plug-in, which pulls camera data directly into Unreal Engine. Real-time lens values drive environments, changing how the visual are presented on screen and creating an accurate and immersive experience for the scene.
“Lighting integration within a virtual production environment is also incredibly important for accurate colour reproduction. Although LED video walls are very good at producing high resolution reflections and a homogenous base level of light, the displays do not provide what we would consider high quality of light due to the narrow spectrums of light emitted, which results in skin tones and fabrics not being faithfully reproduced.” Hence at the company’s Uxbridge stage it has installed a 360-degree ring of light using 50 ARRI Orbiters [LED fixtures] to complement the lighting from the LED walls and provide detailed contrast control and accurate colour reproduction lighting in the volume.”
Refining techniques is also a priority for Brompton. Notes Mead: “We’ve been trying to streamline the process by working closely with other manufacturers producing complementary technologies – motion tracking, cameras and colour grading software, to name a few. A key area of focus for everyone in this space is colour – and how to reliably and repeatedly achieve accurate colour rendition. Brompton’s Dynamic Calibration technology is a great starting point here because we can already deliver accurate colours within an HDR workflow, and we are working with many partners to build on this.”
There is also an awareness that costs need to come down, although as with any new technology area that will surely will happen with the passing of time. Nowak comments: “Camera tracking is progressing, but is still very reliant on stickers; servers and processors are still very expensive; the LED manufacturers are working to develop lower pixel pitch (higher resolution) displays and, while Unreal Engine in considered best-of-breed, it’s still developing to meet the demands of different markets.”
Apart from the technology challenges, it’s also a sector where a new level of collaboration is essential.”You can’t achieve quality, cost efficiency or sustainability without the involvement of the full team,” she adds. “Everyone from the VFX designer to the art department should contribute to the whole production process from the outset. It’s like building a house – you have to start with the technical drawings, you can’t just do it on the fly.”
The expansion of VP has been so rapid that one has to ask the inevitable question: can it last? David ‘Ed’ Edwards is VFX product manager at motion capture specialist Vicon, which recently launched a new motion capture system for the VFX sector, Valkyrie, that is designed to work in conjunction with the company’s VFX software, Shōgun. Edwards remarks: “What I expect now is a period of consolidation which will involve collaboration [between companies] to see how solutions can be simplified and refined.”
This will become increasingly important as – inevitably – film studios look to push VP’s creative limits. “Once people have become confident with a technology it’s natural that they will see what else they can throw at a system,” says Edwards. “So it’s an iterative process with an amount of consolidation, and it means that we are always speaking with customers to inform our long-term product roadmap.”
Meanwhile, many companies are also managing the resurgence of live events such as conferences and concerts (“That whole side has really come back quickly and in a big way,” notes Harding). But here too we can expect the adoption of VP-style techniques as well as blended events that combine in-person and online experiences. This is likely to be especially beneficial for international events such as conferences, where a reduction in air travel has obvious benefits from an environmental perspective at a time when many businesses are defining their decarbonisation strategies.
“While local audiences have mostly returned to live events, virtual production can be used successfully by event organisers to reach international audiences,” says Nowak. “We’re still dealing with the aftermath of the pandemic in terms of travel issues and staff shortages, and streaming events can help alleviate some of the effects of these challenges and promote the creation of content across different platforms. One great example of this is the RealTime Conference: they’ve pretty much made it a staple that they will have an online and live streaming version of the conference and it’s proved very successful.”
Taylor also highlights a crossover between VP and conferencing. “One scenario is that you have a conference in three different venues, but at a certain point you want to a TED Talks-type event where three people appear to be together – well, you can do that with virtual production,” he says, adding that MRMC’s ARC-30 PTZ camera is suitable for deployment across applications.
It’s also likely that a more commoditised and tiered approach to VP provision will emerge as the market matures. “Different packages will become available ranging from the lower to the higher end, but demand will dictate the availability of VP solutions,” says Nowak. “Right now, there’s enormous market interest which is driving technology development from players like disguise, Megapixel VR and Mo-Sys, and increasing the range of solutions available, but accessibility is still an issue because we’re dealing with chip shortages.”
Along with the ongoing supply chain crisis – now widely expected to continue well into 2023 – there is another challenge on the horizon: sufficient availability of trained staff. Encouragingly, a very pleasing aspect to the rise of VP is that many companies are already actively involved in educational efforts of one kind of another, and also see the potential for shaping a more level playing field.
“While we’re in the midst of a skills shortage, virtual production is also creating new roles and opportunities for career development,” says Nowak. “There are no ‘old school’ experts in this field, so there are no obstacles for young people to quickly progress to very senior positions. [In addition] virtual production techniques make the industry more accessible to people with mobility issues who may struggle to work on location and are ideal for parents who can’t afford to be away from home for long periods.”
In terms of how the sector evolves as a whole, it’s clear that adopting an ‘always learning’ approach is the only way to proceed. As Ukas-Bradley notes: “Education on how the technology and workflows are developing will be critical – from higher education to DPs [directors of photography], producers, VFX supervisors and our colleagues in gaming, live events and broadcast worlds.”
For many AV companies who were able to offset some pandemic-related challenges by branching out into VP, it now has every chance of becoming a permanent part of the business, and one that will also have increasing crossovers with other areas of pro-AV and install. And in a period when there is so much to be worried about, that has to be a cause for celebration.