How Deep Learning Enables Large-Scale Scientific Visualization
Using Neural Networks to Compress Information for Interactive Rendering
Introduction
One of the great joys as a Software Architect and Technology Evangelist at Intel is the opportunity to see the fantastic work done by oneAPI Centers of Excellence (CoE) worldwide. Recently, the UC Davis Visualization & Interface Design Innovation (VIDI) Lab shared how they are using deep learning for performant, interactive scientific visualization.
Why does this matter?
Scientific simulations and modeling have become exponentially more detailed in recent years. A simulation can easily generate hundreds of terabytes of data. For simulation models that result in a 3D representation of data, the best way to understand the simulation output is to visually inspect and examine it using a real-time renderer.
As one can imagine, rendering terabytes of data with high-fidelity methods like ray marching (think more advanced ray tracing) and path tracing using even the most powerful GPUs is challenging. A pure, brute-force rendering algorithm and standard compression algorithms don’t provide the performance or compression rates required to support real-time interaction at this scale. While seeing a rendering of simulation output is extremely valuable even at 1 frame per second (fps), providing an actual interactive rendering, say at 30fps, provides a significantly more useful experience for researchers.
This is where the work done by Professor Kwan-Liu Ma and his team: David Bauer and Qi Wu comes in. They combine the latest research in deep learning and rendering to provide something truly remarkable.
Instant Neural Representation for Compression
One of the great challenges mentioned above is compressing data and then quickly decompressing large data sets fast enough to enable a real-time, interactive rendering. If we are looking at rendering at 30fps, the decompression time of simulation data must be less the 33.3 milliseconds.
Compression Types and Rates
The second piece of the puzzle is compression rates. Compression algorithms may be lossless, meaning it is possible to recover the original input data exactly as it was created. They may also be lossy, meaning it is impossible to recreate the original data from the compressed representation. As one may expect, in general, lossless algorithms tend to result in smaller compression rates.
The typical compression that people use every day are things like zip, mp3 and JPEG. The former is lossless and may achieve compression rates of around 3:1. The latter are lossy and typically achieve compression rates of around 20:1 and 10:1. That means the largest compression ratio for these typical algorithms is about 1/20 the size of the original data.
For the purposes of real-time rendering, the main features one would look for are fast decompression and relatively minimal data loss. However, a compression rate of 20:1 may not be enough for many use cases.
Instant Neural Representation to the Rescue
As we’ve seen in the last 10 years, modern neural networks are extremely good at encoding information and decoding it quickly. These two characteristics make neural networks ideally suited as a compression method for large-scale data sets.
Another useful characteristic of using neural networks for compression is that, unlike most compression algorithms, the representation does not grow in size as the data set size grows. One would expect that as data complexity grows, the static representation becomes more lossy.
However, given a large enough network, this degradation is not linear. Because of the huge simulation model sizes, a large neural network may be used which results in good compression ratios and minimal information loss as the data set size grows. The VIDI team reports results that “10–1000x smaller than raw data and trained almost instantaneously using MLP and hash-grid encoding”¹
We can clearly see the value of this compression algorithm versus traditional algorithms in Figure 1.
Creating the Representation
To train the model for compression, the VIDI team had to take the 3D simulation information and convert it into something that fits into a standard machine-learning workflow. Logically this was a relatively standard machine learning flow:
- Grab random samples of data in the 3D space
- Encode each sample using a hash-grid encoding to create the tensor to pass to the DL model
- Train the model for the sample points against the real sample output
To make this as performant as possible, the VIDI team used the Intel® Open Volume Kernel Library, an implementation of the oneAPI OpenVKL specification, to generate training samples from the ground truth for the training stage. OpenVKL is helpful here because it enables efficient traversal and sampling of volumetric data.
With the right configuration, training the model for around 20k steps takes less than a minute. The result of this is a reasonable peak signal-to-noise ratio (PSNR) of around 30dB in most cases and upwards of around 49dB in some datasets. For those unfamiliar with PSNR, it is measured in decibels (dB). The higher the value the closer the image is to the original.
Closing the Rendering Gaps
While Instant Neural Representation is impressive, alone it may not be enough to provide a performant, real-time interactive experience. To bridge this gap, the UC Davis VIDI team also has developed several sampling and rendering algorithms for Instant Neural Representation data.
Unfortunately, the techniques here are outside the scope of this post, but you can check out the details of the Instant Neural Representation and the associated rendering optimizations.
Of course, the more fun thing is to see how all of this fits together to enable a truly interactive experience. You can see in the video below how quickly an Instant Neural Representation can be trained and the impressive results achieved by the UC Davis VIDI team.
Conclusion
I’ve always thought of DL as a technology that can solve extremely complex problems that required a large solution space; I had not considered how it can be used to provide heuristic solutions to problems with “well-known” solutions. The Instant Neural Representation is a great example of taking the power of DL models and using it as a heuristic optimization of some data that can tolerate some level of lossy translation.
Techniques like Instant Neural Representation are enabling useable, real-time interaction with increasingly complex simulation data. Just to emphasize the difference between a 1fps interaction to a 60fps interaction, check out this fun video.
I’m excited and optimistic that novel solutions like this will continue to drive scientific discovery. Enabling a 30+ fps interaction with simulation model data can be a game changer for scientists who will no longer have to live through the slideshow-type interaction with their data.
If you want to see what random tech news I’m reading, you can follow me on Twitter. Also, check out Code Together, an Intel podcast for developers that I host where we talk tech.
Tony is a Software Architect and Technical Evangelist at Intel. He has worked on several software developer tools and most recently led the software engineering team that built the data center platform which enabled Habana’s scalable MLPerf solution.
References
[1] Qi Wu, David Bauer, Michael J. Doyle, Kwan-Liu Ma, Instant Neural Representation for Interactive Volume Rendering, 2022, https://arxiv.org/abs/2207.11620v2
