Casting call for clouds

A new deep learning method for more realistic CGI smoke and cloud effects

Compare any movie from the early nineties with one from 2017, and you’ll see just how far moviemakers have progressed in their use of CGI – computer generated imagery – for realistically simulating either parts of or entire scenes in a movie. Nonetheless, natural-looking smoke and clouds remain among the most difficult effects to simulate. Researchers at the Technical University of Munich (TUM) have now developed a new CGI methodology that promises to be a game changer.

Having worked previously as the R&D lead on movies such as "Iron Man 3" and "Super Man: Man of Steel" at the heart of Hollywood at ScanlineVFX in Los Angeles, Nils Thuerey knows first-hand how time consuming, tedious and expensive this process is. He already has one Technical Oscar Award under his belt for his groundbreaking work in the field of physics simulations for visual effects and CGI. At the TUM, the Professor for Games Engineering works in the field of simulating liquids and gasses, which fall into the general category of fluids.

Rendering the vast amount of detail and the complex motions visible in real-life smoke clouds has been a long-standing challenge for fluid simulations in computer graphics. "Representing the fine features you see in real smoke or clouds typically requires very fine-grained spatial resolutions in a simulation," explains Ph.D. student Mengyu Chu. "This takes up a lot of computational resources and results in very costly simulation runs with painfully long turnaround times in the production process."

Thuerey adds: "Just one simulation for a single smoke cloud can take a day to produce – sometimes longer. And, at the end of the day, if the director doesn’t like what the graphic artist or programmer has produced, you have to throw everything away and start from scratch with a new simulation."


Now Thuerey and Chu have developed a new methodology that introduces deep learning techniques into classical simulation algorithms. Deep learning is part of a broader family of machine learning methods based on learning representations from large amounts of data.

In their method, the algorithm learns from a pre-computed library. There are two versions of the simulations, a "cheap and rough draft version" and a corresponding "finely detailed and expensive one." The algorithm learns which pairs go together. When facing a new situation in another simulation run, their algorithm can then look up the right piece of data from the pre-computed library. Essentially, the deep learning method trains a "smart librarian."


Chu and Thuerey say their vision is to develop a broad library of simulations. "It would allow producers on a small budget to take a cloud or smoke effect that had been created previously and edit it to fit their specific needs," says Chu.

"Right now, our work is most interesting for the movie industry. But our goal is to use it in the future for all kinds of realistic simulation – for example, in medicine," says Thuerey. "I’m particularly interested in blood flow in our bodies. It’s actually the same physics – fluid mechanics – as smoke." For movie studios, the new method could save time and money – but in the emergency room of the future it could prove to be life-saving.

The research is part of the "realFlow – Virtualization of Real Flows for Animation and Simulation" project, for which Thurey received a starting grant from the European Research Council in 2015. The research project aims at the ability to run simulations of physical processes better and, above all, faster.


Mengyu Chu, Nils Thuerey: “Data-Driven Synthesis of Smoke Flows with CNN-based Feature Descriptors”, ACM Transactions on Graphics (SIGGRAPH 2017), Vol. 36 (4), Article No. 69
DOI: 10.1145/3072959.3073643

editorial staff: Erica Gingerich