Dozens of superheroes gathered on a battlefield to face the archvillain Thanos and his army of cybernetic monsters in the finale of Avengers: End Game. By this point in the movie, the culmination of the decade-long storyline woven through 21 films, you're so used to seeing these characters that it's so easy to forget how much of what you see on screen is digitally generated. Portals open in the sky and characters pour into the melee. Lightning crackles around Thor. Energy radiates from Captain Marvel. Valkyrie rides a flying horse. Giant Man towers above the entire mob.

Behind the scenes, teams of artists that outnumbered the entire cast assembled to bring these characters to life. And behind all of the work they did was software, much of it open source.

Software is core to modern filmmaking, from animation and visual effects to editing and post-production. Some of that is handled with well-known commercial software, like Adobe After Effects or Autodesk Maya. But open source software plays an increasingly important behind-the-scenes role in film, animation, and digital effects.

The most visible example is probably the venerable open source 3D-studio application Blender. But there's lots of open source technology under the hood of proprietary applications as well. The Open Shading Language enables artists to programmatically define certain elements of an image, especially the behavior of light. For example, an artist could use it to establish how the Mandalorian’s beskar armor reflects the desert sun of Arvala-7.

The language, originally created by Sony Pictures Imageworks, has been used in films and TV shows like Avengers: Endgame, The Mandalorian, and Soul and now a wide variety of applications, including Blender and products from vendors like Autodesk and Illumination Research, enable artists to use the language within their software without needing to install plugins.

The software used in filmmaking is highly specialized: editors, modelers, and roto artists all use different tools specific to their crafts. Open source is often used to ensure compatibility between these applications as a film moves through production. "Open source has become an important part of how we move work from one piece of software to another throughout the process," says DJ Rahming, Head of Technology at the studio RodeoFX. "Open standards and libraries give us a way to define things, like textures for example, and have a common way of working with them throughout the industry." Without these standards, one artist’s work might look entirely different when opened by an artist using another application.

One of the most widely used open source technologies in the modern film industry isOpenColorIO, a color management system that makes it possible to consistently define color across different applications and projects. That may sound simple, but it's an important factor in ensuring consistency. It's part of what ensures that Mile Morales’s costume is always the same color, scene after scene, and gives the alien worlds of the Star Wars films and shows their vivid colors.

"Defining what different color components actually mean is a problem everyone in the industry needs to solve," Rahming says. "OpenColorIO finally makes that possible."

These standards can save studios from tedious and error-prone work. For exampleOpenTimelineIO, open sourced by Pixar in 2016, helps ensure that a collection of video edits created in one application will work in other applications that support the standard. "Before OpenTimelineIO we had to keep track of edits by having a production assistant manually enter time codes for different shots into a database," Rahming says.

Why Studios Are Opening Up

Commercial software alone can't meet the needs of animation and visual effects studios like Industrial Light and Magic, Rodeo FX, and Weta. "When we started doing digital visual effects, there wasn't any software, commercial or otherwise, that did what we wanted," says industry veteran Larry Gritz, a software engineering architect at Sony Pictures Imageworks, and the biggest contributor to Open Shading Language. "Most of what we used was built in-house."

Even as more commercial solutions entered the market, studios kept building their own software. "Doing things you've never seen before is one of the hallmarks of the industry," says Kimball Thurston, Head of Engineering at the visual effects company Weta Digital and chair of the technical advisory council of the Academy Software Foundation (ASWF), an umbrella group for open source software used by the film industry. "We're constantly pushing the boundaries of what's possible. To do that, we can't turn to off-the-shelf software. We have to write our own."

For example, DreamWorks Animation created OpenVDB, a library and data format for describing "volumetric" information—a task essential for representing things like smoke, water, and mist. Think of the smoke and fire billowing out of an explosion, or imagine the splash of water when an alien ship crashes into a large body of water.  OpenVDB makes it possible to animate these things with the realism modern audiences expect. "Dreamworks was really pushing the limits of what was possible at the time," Thurston says.

Though studios are in constant competition, trying to one-up each other in the quest to create the most breathtaking experiences possible for audiences, they also cooperate across company lines. A single film might have more than a dozen different effects studios working on it. "Production companies farm work out to lots of different companies," ILM R&D Supervisor and ASWF Technical Advisory Council Member Cary Phillips says. "So even if we compete for contracts, we're also collaborating with other studios on projects."

This is also why compatibility is critical in the industry. Teams need to hand off massive files not only to other applications and departments, but also to different studios and cities. Studios still keep many of their innovations proprietary, but they find that open standards, coupled with open implementations help make the entire production more efficient. Sharing software can also ensure that the film industry benefits from its technical talent. "Sometimes we have trouble attracting top-tier computer science people to the industry," Thurston says. "Not everyone recognizes that there are interesting challenges to be solved in this field, so it helps to spread the workload around."

Open standards can also help studios avoid being locked into specific products or vendors, says Peter Hillman, a Senior Software Engineer at Weta Digital. But more importantly, it helps studios create their own software. "We have many in-house tools, including our own renderer, and we use open source libraries to support common standards," he says. "It saves us a lot of work."

The Foundations of a More Open Film Industry

ILM helped start the industry's open source revolution when it released their implementation of the OpenEXR image format in 2004. "OpenEXR really paved the way for that idea of open source in the film industry," Gritz says. "Now all the studios are working with technologies developed by cross-industry teams. It's hard to imagine working any other way."

The origins of the OpenEXR format date back to 1999. Studios store each frame of a video as a discrete file, which enables them to process many frames in parallel, as opposed to one at a time, enabling them to render work more quickly, explains Hillman. Eventually, those frames are exported into video formats, but the vast majority of post-production work on a film is done with a series of still images.

When ILM developed what eventually became the OpenEXR format, lossless image formats like TIFF only supported a fixed number of color channels, which limited their utility in a production pipeline. OpenEXR supported an unlimited number of channels, enabling studios to capture images in far more detail than ever before. "The major innovation was using floating-point numbers to represent image intensities, rather than fixed integers," Phillips explains. "That made it possible to represent a much wider range of colors while keeping file sizes relatively small."

The format also supports unlimited metadata types, which means developers can create software that adds any sort of information they want to a file, like information for creating a photo blur for instance, without worrying about breaking compatibility with other software.

OpenEXR remained a proprietary format at ILM for five years before the company decided to open source it. When they did, they not only released the specification but a reference implementation—a library that developers could use to integrate OpenEXR into their own software.

"At ILM we use dozens of different software packages from all sorts of vendors, along with tools we create ourselves," Phillips says. They eventually realized it made more sense to share the format with the rest of the industry, rather than keeping it proprietary. That way third-party vendors could easily add OpenEXR support to their products and partner studios could work in the same format ILM uses.

"If they had kept it to themselves, OpenEXR would have been less useful to ILM internally than it is now that everyone can use it," Hillman says.

Opening the format worked out well. Today, practically all video software packages support the OpenEXR format natively—as do the Microsoft Windows and MacOS operating systems. "I'm reasonably confident that every movie made in the last several years uses OpenEXR," says Thurston.

By 2013, other studios were collaborating to make OpenEXR better. Weta added support for "deep data," enabling the format to store more than one piece of color information for each pixel. That's useful for creating images such as objects cloaked in a layer of fog. Dreamworks contributed a new image compression algorithm.

Gradually other studios followed suit. In 2010, Sony Pictures Imageworks open sourced the OpenColorIO standard, followed by the Open Shading Language in 2012. That same year DreamWorks Animation released OpenVDB.

The rapid adoption of open source, particularly OpenEXR and OpenColorIO, created pressure on vendors to support these emerging standards. "At a certain point, vendors had to support technologies," Rahming explains. "If you don't support OpenColorIO, for instance, it's going to be a deal breaker for studios."

By 2017 it was clear that open source was vital to the health of the industry. Many stakeholders, including vendors and studios, began talking about how to sustain the continued development of key shared assets. The original developers of OpenEXR had all left ILM by that point, so the companies saw a need to find a more cooperative approach to funding its continued development. Even as studios reaped the benefits of inter-company collaboration, the idea of paying developers to work on software technically owned by another company was awkward, even if that software was open source. "We knew that if we wanted to keep maintaining these projects we should put them under a neutral entity to ensure long-term contributions and collaboration from a wider community," Thurston says.

After extensive discussions, The Academy Software Foundation was founded in 2018 under the umbrella of the Academy of Motion Picture Arts—the same Academy behind the Academy Awards—and the Linux Foundation. "The Academy Software Foundation's mission is to ensure the health of the software our industry depends on," Phillips says.

The ASWF is still adding new projects. Apart from Blender, much of what the industry has open sourced thus far are building blocks for creating software. Thurston hopes studios will soon release more tools for artists in addition to those they create for developers.

"We're in this because we're movie fans," Phillips says "We want to see new things on the screen. Open source helps us do that."