Skip to main content

Virtual production gets real: Bringing real-time visual effects onto the set

Digital tools andtechnologies are bringing virtual effects to physical sets and makingproduction more flexible and cost-effective—freeing creativity from previous constraints.

Jeff Loucks
Chris Arkenberg
Kevin Westcott

The tools and techniques of virtual production are steadily transforming film and cinema production, increasing flexibility, shortening production times, and bringing real-time computer-generated imagery (CGI) and visual effects out of postproduction and onto real-life sets. Deloitte Global predicts that the market for virtual production tools will grow to US$2.2 billion in 2023—up 20% from an estimated US$1.8 billion in 2022. Multiple forces are driving this growth: audiences demanding film and television genres that rely heavily on digital capabilities; streaming video services embracing these genres to fill production slates and reduce costs; greater reliance on digital tools as workforces become more remote and distributed; and the ambitions of some top game engine providers to better serve the film and video production market.

Virtual production brings the impossible to the set

CGI and visual effects have been enriching cinematic storytelling for decades. The 2001 cinematic adaptation of J.R.R. Tolkien’s The Lord of the Rings was an example of how these tools and techniques came together to push the boundaries of visual storytelling.

However, behind the scenes, there was a disconnect between what was possible during on-set production and what had to be postproduced with software. Actors performing in front of green screens were prompted to imagine the sets and effects that would be added afterward. Directors and cinematographers could see a person in a motion-capture suit on set but not the 3D-rendered creature they would become on screen. Fantastic settings couldn’t be visualized until postproduction. Moreover, many shots were on location, requiring studios to fly crews and gear to the site, contend with the whims of local conditions, and incur high costs in money, time, and uncertainty. As more studios entered the market and pressures to reduce costs and time-to-screen mounted, filmmakers became increasingly constrained by these limitations.

Virtual production offers greater freedom and flexibility. Using technologies such as game engines, LED volumes, and augmented reality, virtual production can bring real-time computer-generated graphics and visual effects directly into the production process, enabling everyone on set to see and interact with them. Sets can be built inside a game engine or captured using photogrammetry—for example, by scanning the surfaces of a Paris neighborhood to create a 3D model. The virtual set and its effects can then be rendered in high-resolution on LED volumes—walls and ceilings made of LED screens that surround the physical set—transforming a sound stage in Burbank into that Paris neighborhood. Actors can see digital sets and elements and react to them more naturally. Cameras with augmented reality layers let operators and cinematographers see more dynamic digital assets such as virtual characters driven by motion-capture actors. Stage cameras can align with virtual cameras to precisely track movements across physical and virtual spaces.

And virtual production’s benefits can extend far beyond what it enables on set. For many producers, it’s an invaluable tool for making the filmmaking process much more flexible. For example, objects in the digital set can be repositioned on demand, and lighting between real and virtual elements can be more easily matched. Production time can be shortened, costs can be reduced, time and location constraints can be removed, and creativity can become more unbounded.1

As studios support more streaming video services—or launch their own—many highly competitive production pipelines are taking advantage of virtual production. For instance, using virtual production, ILM turned a few soundstages and a backlot into 2019’s The Mandalorian, bringing an alien universe with cinema-quality effects to streaming TV.2 The show’s success has been cited as a catalyst for virtual-production adoption.3

There’s certainly no shortage of demand for content with spectacular—and often otherworldly—visuals. Over the past decade, adventure and action movies have become the most successful film genres, followed closely by fantasy and sci fi. Such stories are a perfect stage for CGI and visual effects, and they increasingly require virtual production capabilities that enable the impossible and transport actors and audiences into other worlds.4 Leading streaming video services are also filling their programming with these genres, with virtual production being central to developing effects and experiences that satisfy their demanding audiences.5

The COVID-19 pandemic helped underscore virtual production’s value and boosted its adoption.6 Closures and social distancing shifted audiences from theaters and mobile to in-home viewing just as leading media and entertainment providers were launching their streaming video services. The need for streamers and studios to fill their catalogs with original content without sacrificing quantity or quality pushed studios toward virtual production and visual effects, especially for the more popular visual effects–rich genres. The pandemic also required mostly colocated production teams to adopt remote and distributed workflows. Instead of flying people to filming locations or bringing them together on sets, studios turned to virtual production solutions that could replicate those production elements.

Advances in developing hyperrealistic video games have been a key enabler of virtual production. The Unreal Engine from Epic Games has been recognized as a pioneer in using high-performance computing, 3D modeling, and physics to enable the shift from laborious and time-consuming postproduction to photorealistic and real-time rendering on-set—advancing their own games’ capabilities in the process.7 Other game engine leaders, like Unity Technologies, are also enabling the convergence of 3D gaming experiences and 2D storytelling, which will likely spark more movement of intellectual property (IP), audiences, and entertainment between them. On the way to the promised land of the metaverse, the collaboration between film and gaming could unleash new forms of entertainment that mix cinematic storytelling, immersive and social gaming, and real-time broadcasts.

The bottom line

Virtual production has already gained a strong foothold in film and TV studios and is poised to move from early adoption to early majority.8 The flexibility and freedom that it offers storytellers and production crews are a spur to its increasing use. And with competition among streamers heating up and audiences facing a possible recession, cost pressures on content development are likely to become stronger, prompting more studies to rely on virtual production to reduce costs and time to market.

One potential headwind, however, is that virtual production itself represents a significant investment. Even though—if planned well—it can cost less and be quicker than traditional techniques, the tools involved can be complicated and difficult to use, requiring significant training as well as sophisticated and expensive hardware. Qualified virtual production talent is currently scarce—and therefore costly. In addition, the landscape of bespoke providers is fragmented.9 Larger and more established studios have assembled their own virtual production toolchains and are learning how best to integrate them into production—but smaller studios and streamers moving into CGI-based entertainment may instead partner with third-party providers that can provide the talent, software, and hardware in the same package. All these users will likely reckon with uneven standards, setup and tracking challenges, and the nuances of seamlessly blending the physical and digital.10

Virtual production also affects how productions are planned and financed. Because it shifts much of the work that was done in postproduction into the preproduction phase, studios must do more up-front work to build digital assets, match colors, and properly set up hardware and software. This can deter investors who balk at having to provide funding earlier in production.

Over time, however, the cost, expertise, and funding barriers will likely come down as the virtual production market grows. And as the industry matures and best practices are discovered, virtual production is becoming easier to use. The interplay of gaming and video production could potentially generate a feedback loop of content development as digital assets like sets, characters, and imagery can move more easily between mediums. More studios are pursuing franchises built around “universes” that can be experienced through video and gaming—and more audiences are becoming used to engaging with rich, imaginative, and hyperrealistic virtual worlds. As entertainment becomes increasingly social and interactive, the technologies of virtual production will likely be seen as fundamental building blocks and enablers of the emerging metaverse.

Show more

Technology consulting

Innovation, transformation, and leadership occur in many ways. The ability to solve complex issues is critical. Together, we can help you imagine, deliver, and run your business, wherever you compete, using the latest technologies like cloud and cognitive, from strategy development through implementation.

Learn More

The authors would like to thank Duncan Stewart and Ankit Dhameja for their help in supporting this research.

Cover image by: Jaime Austin and Sofia Segi

Did you find this useful?

Thanks for your feedback

If you would like to help improve further, please complete a 3-minute survey