Learn · Intermediate

NeRF and Gaussian splatting: turning photographs into a scene you can move through

NeRF and Gaussian splatting are two ways of turning a set of ordinary photographs into a three-dimensional scene you can view from angles no camera ever occupied. A NeRF stores the scene inside a small neural network that answers, for any point in space and any viewing direction, what color you see there and how solid the space is. Gaussian splatting stores the same information as millions of tiny translucent ellipsoids. Both learn by rendering a guess, comparing it to the real photographs, and adjusting.

What came before, and why it fell short

Classical photogrammetry finds matching features across photographs, triangulates them into a point cloud, and wraps a triangle mesh around it. It works, and it is what most 3D scanning has run on for twenty years.

It also fails on exactly the things that make a scene look real. A mesh is a hard surface, so glass, smoke, fur, foliage, and thin structures like wire fences either get approximated into mush or vanish. And a mesh with a fixed texture cannot represent view-dependent appearance: the highlight that slides across a car's hood as you walk past it is a property of the direction you are looking from, not of the paint.

The NeRF idea

In 2020, Ben Mildenhall and colleagues proposed something that sounds too simple to work. Represent the entire scene as one small neural network. Feed it a 3D position and a viewing direction. Have it output a color and a density.

To render a pixel, march a ray out from the camera through that pixel, sample points along it, ask the network what is at each one, and composite the results front to back, weighting each sample by how much light it blocks. That compositing step is differentiable, which is the crucial property: you can render a pixel, compare it to the real photograph, and push the error backwards into the network's weights with ordinary gradient descent.

There is no 3D supervision anywhere. Nobody tells the network where the surfaces are. It discovers geometry because that is the only way to make renderings from dozens of different viewpoints agree with each other simultaneously.

One detail from the original paper is worth knowing because it recurs everywhere. Feeding raw coordinates into the network produced blurry results, so the authors first mapped each coordinate through a bank of sine and cosine functions at many frequencies. That is the same positional encoding trick transformers use on token positions, and for the same reason: neural networks are biased toward smooth functions, and expanding the input into multiple frequencies makes fine detail learnable.

The catch was speed. Hundreds of network queries per pixel meant days to train one scene and seconds to render one frame.

The splatting idea

In 2023, Bernhard Kerbl and colleagues asked whether the neural network was necessary at all. Their answer, largely, was no.

3D Gaussian splatting represents the scene as a large collection of 3D Gaussians, each an ellipsoidal blob with a position, a shape and orientation, an opacity, and a color that varies with viewing angle. Rendering means projecting each blob onto the image plane and blending them in depth order. That is a rasterization problem, which is precisely what graphics hardware was built for, and it runs in real time.

Training works the same way as NeRF conceptually, but the optimization now moves explicit objects rather than network weights. Start from a sparse point cloud, and repeatedly render, compare to the photographs, and adjust each blob's position, shape, opacity, and color. The system also splits blobs that are trying to cover too much detail and deletes ones that have become transparent, so the representation grows where the scene is complicated and thins out where it is not.

Two useful analogies. A NeRF is like a formula you evaluate to find out what is at a location: compact, and slow because you evaluate it constantly. Gaussian splatting is a lookup table of overlapping soft brushstrokes: much larger on disk, and fast because reading it is trivial. Impressionist painting is the closer visual analogy for splatting, where the scene emerges from thousands of individually meaningless dabs.

The tradeoff is honest. Splatting files are large, and the representation has no explicit surfaces, which makes extracting a clean mesh for physics or 3D printing awkward. Related work like Plenoxels showed the same lesson from another angle: much of NeRF's quality came from the differentiable rendering and the optimization, not from the neural network specifically.

Moving scenes, and where this connects

Extending either method to time gives you 4D: three spatial dimensions plus motion. The usual approach adds a deformation field that says how the static scene bends at each moment, so you learn one canonical scene plus how it moves rather than a separate scene per frame.

That is the capability generative video work has been converging on, because a video model that can be read for persistent geometry is much closer to a simulator than to a frame painter. This week researchers showed that one checkpoint can turn a compatible video model into a 4D world builder by feeding the video model's internal representation directly into a 4D decoder, skipping pixels entirely. The connection to world models is the point: a system that holds a scene rather than a sequence of images can be queried, navigated, and used to predict.

What to remember

Both methods learn 3D structure with no 3D supervision, purely by forcing renderings from many viewpoints to agree. NeRF proved the idea was possible; Gaussian splatting made it fast enough to use. The choice between them is a compact-and-slow versus large-and-fast tradeoff, and for most practical work today, splatting wins.

Key papers
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
3D Gaussian Splatting for Real-Time Radiance Field Rendering
Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
Plenoxels: Radiance Fields without Neural Networks
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering

Key questions

What problem do NeRF and Gaussian splatting solve?

Both reconstruct a full three-dimensional scene from a set of ordinary photographs, so you can render views from camera positions that never existed. Traditional photogrammetry produces a mesh that struggles with glass, smoke, fur, and reflections, while these methods represent the scene as something continuous and handle those cases naturally.

How is Gaussian splatting different from NeRF?

NeRF stores the scene inside a neural network that must be queried hundreds of times per pixel, which is slow. Gaussian splatting stores it as millions of explicit translucent ellipsoids that can be projected and blended directly by a graphics card, making it roughly two orders of magnitude faster to render at comparable quality.

Do these methods need special equipment?

No. Both learn from ordinary photographs or video frames, provided the camera positions are known or can be recovered by standard structure-from-motion software. No depth sensor or lidar is required, though having one helps in texture-poor scenes.

Cite this

APA

Ground Truth. (2026, August 12). NeRF and Gaussian splatting: turning photographs into a scene you can move through. Ground Truth. https://groundtruth.day/learn/nerf-and-gaussian-splatting.html

BibTeX

@misc{groundtruth:nerf-and-gaussian-splatting,
  title  = {NeRF and Gaussian splatting: turning photographs into a scene you can move through},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/nerf-and-gaussian-splatting.html}
}

Topics: computer-vision · 3d · rendering · world-models · graphics