Mixed reality entertainment, encompassing VR, AR, motion capture, live CG graphics, and real, in-person performance, is growing in popularity. Entirely virtual worlds inside VR headsets are now possible, with live, on-stage performers generated using LED Walls. Used to its full potential, mixed reality entertainment offers different performance genres the opportunity to expand beyond their traditional limitations.
Virtual production and mixed reality technology have seen a boom in recent years. Even prior to the pandemic and the resulting lockdown restrictions, we saw the beginnings of an amalgamation of different entertainment technologies, as studios attempted to enhance viewer experiences and provide new perspectives on established genres.
Take for instance the National Theatre’s Draw Me Close production in 2019, placing members of the audience in virtual reality headsets while actors interacted in animated form using motion capture suits to drive realistic body movements. Or, Madonna’s augmented reality performance that utilised volumetric capture and the Unreal Engine to generate multiple on-stage counterparts of the pop star as she performed.
At Shocap Entertainment, we too had already begun working on ways to bridge the incredible technologies available to Hollywood and triple-A game production, partnering with The 7 Fingers, a performance art and circus collective, to produce live circus and theatre that fused on-stage CGI, virtual lighting and motion capture.
CENTRE STAGE
There’s no question the events of 2020 have positioned these ideas centre stage. As creators in the entertainment industry looked to adapt and find new audiences outside of traditional frameworks, performances that could implement modern immersive technology successfully offered viewers unique experiences without needing to be physically present. What’s interesting is the technological adaptations that look set to stay – even as industries begin returning to varying levels of normality.
Arguably, one of the most notable examples was Travis Scott’s virtual appearance inside the hugely popular video game, Fortnite. This was notable for two key reasons: one, it provided a literal crossover of the popular music and video game industries with a remote audience, and two, it was all generated using a video game engine. This is important and feeds into the industry trends for mixed reality performance we’re seeing establish themselves in 2021.
UNREAL PLAUDITS
Epic Games’ Unreal Engine is receiving wide applause across the media and entertainment world, and for good reason – it’s the driving force behind the industry shift towards virtual production. It connects different hardware solutions, generating realistic CG visuals that can be displayed on large LED walls, inside virtual reality headsets, and on traditional screens.
It’s well beyond the categorisation of ‘what’s to come’ and is already being implemented across the industry successfully. It’s the basis for a range of mixed reality events and is being used regularly in broadcasted media, generating set extensions, entire virtual studios, and interactive graphics to enhance viewer engagement.
Now the technology is here, studios will be looking to find meaningful ways for audiences to engage with performers, and each other, during the show. Interactive elements and game-like mechanics that enable audiences to directly add or alter the content of the scene in real-time can add a lot of excitement to an event, especially when those elements reward audiences for collective behaviour.
TRUE ENTERTAINMENT
A live, mixed reality event is not simply an in-person event that you live stream. Pointing a camera at a performer and doing nothing else reinforces the feeling of disconnectedness and isolation, which is the exact opposite of a truly immersive performance. This is at the heart of our own thinking when working on projects with The 7 Fingers – we’re continually looking at ways to use computer-generated images in ways that add new perspectives to the in-person event rather than just for show.
True mixed reality entertainment has the potential to transform in-person and remote viewership in ways that don’t detract from the feeling of excitement audiences expect from in-person events. The underlying concept is to create shows that are live and virtual – both in a physical venue and outside of it.
With modern, mixed reality technology, studios and creative facilities can physically generate fictional worlds that take place within an on-stage character’s imagination, positioning a virtual camera on the fantastical. Real performers can interact with the virtual performers on stage – this is how mixed reality can enhance live entertainment in unique and exciting ways.
