The Return of the Blob: How a Fuzzy Geometric Primitive is Revolutionizing 3D Graphics
Creating realistic, interactive 3D worlds from just a handful of viewpoints has been a holy grail in computer vision for decades. The promise of letting computers “see” akin to how we humans do. This capability unlocks transformative applications, from enhanced virtual reality to robotic perception.
In recent years, deep learning, the turbocharged form of machine learning that utilizes massive datasets to train algorithms commonly associated with neural networks, has made remarkable progress in AI. It seemed destined to revolutionize this field as well.
However, most early deep learning methods focused on meshes—collections of interconnected triangles that define a 3D object’s surfaces.
While useful for smooth surfaces and consistent lighting, meshes struggle to capture the intricate details and nuance of the real world where reflectivity, opacity, and color change depending on the viewing angle (a phenomenon known as view-dependent effects.)
Enter the forgotten hero: the “splat.” These adaptable, fuzzy blobs, more formally known as 3D Gaussian splats, are a point-based rendering technique that predates deep learning. While unpopular for two decades, they’re making a comeback in a unique way.
### From Static Splats to
Researchers experimenting with meshes and deep learning hit a novel idea, recognizing that they needed a new geometric primitive that could gracefully handle intricate details like light reflecting off a coffee cup or the undulating texture of fabric.
Their
Splashes in AI Training
One method, Neural Radiance Fields (NeRFs) offered a phased approach by leapfrogging over the limitations of meshes, using neural networks to represent an appearance function of a scene, representing 3D color information without requiring detailed geometric data. This was revolutionary, yielding photorealistic novel-views from just a handful of input images.
NeRFs spurred significant advancements, but they posed their own challenges. They are computationally expensive to train, requiring hours on powerful GPUs to learn a single scene. Real-time rendering with NeRFs is still in its infancy, making them unsuitable for interactive experiences like video games or augmented reality.
Research continued to push the boundaries, leading to more efficient variations of NeRFs. However, one group took a different tack entirely. They knew there had to be a more direct and efficient solution for representing scenes.
Enter the return of the blob,
### Injecting Simplicity
George Drettakis, a veteran in computer graphics and a leading researcher at INRIA, recognized the enduring appeal of point-based rendering techniques.
“It seemed to me that the leverage point-cloud representations offered a simpler path to photorealistic rendering, particularly for complex scenes with nuanced lighting,” Drettakis explains.
Drettakis and his team, including his postdoc Bernhard Kerbl and PhD student Giorgios Kopanas, had several key advantages: decades of deep knowledge about GPUs and rendering techniques.
Drettakis saw an opportunity and seized it, aiming to combine point-based rendering with machine learning techniques. The result was a new approach using 3D Gaussian splats to represent scenes efficiently capturing fine details and lighting effects in a compact and easy-to-translate format
صبحت 3DGaussian splats uniquely capable of capturing the intricate details of a scene by storing visual details directly within each point.
More importantly, they wouldn’t be built on the complex architecture of neural networks. The researchers carefully designed algorithms mimicking key aspects of machine learning, leveraging techniques like gradient descent and loss functions, resulting in faster
3D Gaussian splats were suddenly not just a
