Published Sep 21, 2026 - Author: Guoxiang (Shawn) Zhou
Two methods made photorealistic 3D reconstruction from ordinary photos practical: NeRF (Neural Radiance Fields, 2020) and 3D Gaussian Splatting (3DGS, 2023). Both take photos with known camera poses, fit a radiance field to them, and render the scene from viewpoints nobody photographed. They render with the same equation and differ in one choice: NeRF parameterizes the radiance field densely with a neural network, Gaussian splatting sparsely with millions of explicit Gaussians. That one choice decides speed, memory, editability and how well each fits on a robot.
Both build on two ideas covered in my post on differentiable rendering: fit a scene representation to images by gradient descent through a differentiable renderer, and render a radiance field (density and view-dependent color at every point) with volume rendering. The one equation to keep in mind is the discrete volume rendering formula, which composites the samples along a camera ray front to back:
\[C = \sum_{i} T_i\, \alpha_i\, \mathbf{c}_i, \qquad T_i = \prod_{j=1}^{i-1} (1 - \alpha_j)\]where $\alpha_i$ is the opacity of sample $i$, $\mathbf{c}_i$ its color and $T_i$ the transmittance, the fraction of light that reaches it. NeRF evaluates a neural network at hundreds of samples per ray to get $\alpha_i$ and $\mathbf{c}_i$. Gaussian splatting reads them off a set of explicit Gaussians, which is what makes it fast.
NeRF (Mildenhall et al., ECCV 2020) parameterizes the radiance field densely, at every point in space, with an MLP, and renders it with volume rendering.
The network maps a position and a viewing direction to a density and a color:
\[F_\theta: (\mathbf{x}, \mathbf{d}) \mapsto (\sigma, \mathbf{c})\]The architecture enforces the halfway fix for view-dependent color. The position goes through 8 fully connected layers of 256 channels (with a skip connection that re-injects the input at the fifth layer), which output the density $\sigma$ and a feature vector. Only then is the viewing direction concatenated, and one more layer of 128 channels outputs the color. So the density, which is the geometry, cannot depend on the viewing direction, which forces the geometry to be consistent across views, while the color can.
Fed raw coordinates, an MLP produces blurry results. Neural networks have a spectral bias towards smooth, low-frequency functions, which is the neural tangent kernel view of an MLP. NeRF fixes this by mapping each coordinate to sines and cosines of increasing frequency before the network:
\[\gamma(p) = \big(\sin(2^0 \pi p), \cos(2^0 \pi p), \dots, \sin(2^{L-1} \pi p), \cos(2^{L-1} \pi p)\big)\]with $L = 10$ frequencies for the position and $L = 4$ for the direction. Nearby points that were almost identical as inputs now differ strongly in the high-frequency components, so the network can represent sharp edges and fine texture. Tancik et al. (2020) later explained this as changing the NTK from a low-pass kernel into one with a tunable bandwidth.
NeRF is fitted to one scene at a time; there is no pretraining and no generalization to other scenes. Each training step:
The original NeRF uses no depth at all; depth supervision from LiDAR or depth cameras is an optional extra that speeds up training and improves geometry. The samples along each ray are stratified: the range $[t_n, t_f]$ is split into $N$ equal bins and one point is drawn uniformly inside each bin. Since the sample positions change every iteration, the network sees the whole continuous ray over training rather than a fixed set of points.
import torch
def render_rays(field, rays_o, rays_d, near, far, n_samples=64):
"""Volume-render a batch of rays. rays_o, rays_d: (B, 3), rays_d of unit length."""
bins = torch.linspace(near, far, n_samples + 1)
lower, upper = bins[:-1], bins[1:]
t = lower + (upper - lower) * torch.rand(rays_o.shape[0], n_samples) # stratified, (B, N)
pts = rays_o[:, None, :] + t[..., None] * rays_d[:, None, :] # (B, N, 3)
sigma, rgb = field(pts, rays_d[:, None, :].expand_as(pts)) # (B, N), (B, N, 3)
delta = torch.diff(t, dim=-1)
delta = torch.cat([delta, torch.full_like(delta[:, :1], 1e10)], dim=-1)
alpha = 1.0 - torch.exp(-sigma * delta)
# T_i = prod_{j<i} (1 - alpha_j), an exclusive cumulative product.
ones = torch.ones_like(alpha[:, :1])
trans = torch.cumprod(torch.cat([ones, 1.0 - alpha + 1e-10], dim=-1), dim=-1)[:, :-1]
weights = trans * alpha
color = (weights[..., None] * rgb).sum(dim=1)
depth = (weights * t).sum(dim=1)
return color, depth, weights
for step in range(n_steps):
rays_o, rays_d, target = sample_random_rays(images, poses, batch_size=4096)
color, _, _ = render_rays(nerf, rays_o, rays_d, near, far)
loss = ((color - target) ** 2).mean()
optimizer.zero_grad()
loss.backward()
optimizer.step()
Most samples along a ray land in empty space or behind the first surface, where they contribute nothing. NeRF trains two networks at once to spend samples where they matter:
Both networks are trained with the photometric loss. This is importance sampling: the integral is estimated with samples concentrated where the integrand is large.
The original NeRF produced a breakthrough in quality, with only about 5 MB of weights per scene, but it has clear limitations:
A large body of follow-up work addresses these:
3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) renders in real time at a quality equal to or better than the best NeRFs, and trains in tens of minutes. It keeps the volume rendering equation, but replaces NeRF’s dense neural field with an explicit, sparse one.
NeRF parameterizes the radiance field everywhere, yet most of space is empty. Gaussian splatting parameterizes it only where the density is nonzero: the scene is a set of millions of 3D Gaussian blobs floating in space, each with its own position, shape, opacity and color. Space between the blobs has zero density and costs nothing to render. And because each Gaussian has a closed-form shape, we know directly which Gaussians a pixel’s ray passes through, instead of blindly sampling points along the ray and asking a network what is there.
Each Gaussian $i$ is a 3D density with mean $\boldsymbol{\mu}_i$ and covariance $\Sigma_i$:
\[G_i(\mathbf{x}) = \exp\left(-\tfrac{1}{2} (\mathbf{x} - \boldsymbol{\mu}_i)^T \Sigma_i^{-1} (\mathbf{x} - \boldsymbol{\mu}_i)\right)\]The covariance makes the Gaussian anisotropic: a flat disc can cover a wall with one Gaussian, and a long needle can cover a thin edge. But $\Sigma$ cannot be optimized directly, because gradient descent would quickly make it non-positive-definite, which is not a valid covariance. So the covariance is built from a rotation $R$ and a diagonal scaling matrix $S$:
\[\Sigma = R\, S\, S^T R^T\]which is the eigendecomposition of $\Sigma$ in reverse: $R$ holds the principal axes and the scales $s_k$ are the standard deviations along them. Any rotation and any scales give a valid covariance. The figure shows the 2D version, where $R$ is a rotation by an angle.
In 3D, the rotation is stored as a unit quaternion. In total, each Gaussian has 59 parameters:
| Parameter | Numbers | Activation |
|---|---|---|
| Position $\boldsymbol{\mu}$ | 3 | none |
| Rotation (quaternion $\mathbf{q}$) | 4 | normalized to unit length |
| Scale $\mathbf{s}$ | 3 | exponential, so scales stay positive |
| Opacity $o$ | 1 | sigmoid, so it stays in $(0, 1)$ |
| Color (spherical harmonics, degree 3) | 48 | none |
The color must depend on the viewing direction $\mathbf{d}$ to show highlights and reflections. Instead of a network, each Gaussian stores coefficients of spherical harmonics $Y_l^m$, the Fourier basis for functions on a sphere:
\[\mathbf{c}_i(\mathbf{d}) = \sum_{l=0}^{3} \sum_{m=-l}^{l} \mathbf{k}_{i,l}^m\, Y_l^m(\mathbf{d})\]Degree 0 is a single constant, the diffuse color. Each higher degree adds finer angular detail. Degree 3 has $1 + 3 + 5 + 7 = 16$ basis functions per color channel, so $3 \times 16 = 48$ numbers. Training starts with degree 0 only and unlocks one more degree every 1000 iterations, so the optimizer gets the base color right before it explains the remaining error with view dependence.
Projection. A 3D Gaussian seen through a perspective camera is approximately a 2D Gaussian on the image, a technique called EWA splatting (Zwicker et al., 2001). The mean projects like any point. The covariance is transformed with the camera rotation $W$ and with $J$, the Jacobian of the perspective projection linearized at the Gaussian’s center:
\[\Sigma' = J\, W\, \Sigma\, W^T J^T\]The top-left $2 \times 2$ block of $\Sigma’$ is the covariance of the 2D Gaussian on the image. The linearization is the approximation: it is exact for the center and good as long as the Gaussian is small compared with its distance from the camera.
Blending. For a pixel $\mathbf{p}$, the opacity of Gaussian $i$ is its learned opacity times its 2D Gaussian evaluated at the pixel:
\[\alpha_i = o_i \exp\left(-\tfrac{1}{2} (\mathbf{p} - \boldsymbol{\mu}'_i)^T \Sigma_i'^{-1} (\mathbf{p} - \boldsymbol{\mu}'_i)\right)\]and with the Gaussians that overlap the pixel sorted by depth, front to back, the pixel color is
\[C = \sum_{i} \mathbf{c}_i\, \alpha_i \prod_{j=1}^{i-1} (1 - \alpha_j)\]This is the same formula as NeRF’s volume rendering. The only difference is where $\alpha_i$ comes from: in NeRF it is the density of a sampled point integrated over a ray segment, in Gaussian splatting it is read off a closed-form 2D Gaussian. The volume rendering integral is computed without ever sampling a single 3D point in space.
One approximation is hidden in the sort: each Gaussian gets a single depth, that of its center. When two Gaussians overlap in depth, their order can flip as the camera moves, which can cause “popping”. Later work (StopThePop, 3D Gaussian ray tracing) sorts per pixel or traces rays to remove it.
The renderer is what makes Gaussian splatting fast. It is a custom CUDA rasterizer, not a ray tracer:
Click a tile to see its sorted list.
The backward pass for training walks the same lists back to front. Because each pixel only stored its final transmittance and the number of Gaussians it used, the intermediate transmittances can be recovered by dividing out each $(1 - \alpha_j)$, so memory stays small.
Training follows the differentiable rendering loop:
Compute the loss against the photo, a mix of L1 error and a structural similarity term:
\[\mathcal{L} = (1 - \lambda)\, \mathcal{L}_1 + \lambda\, \mathcal{L}_{\text{D-SSIM}}, \qquad \lambda = 0.2\]The default schedule runs 30,000 iterations. The main difficulty is that this optimization is badly non-convex: a Gaussian cannot “jump” to where it is needed, it can only slide along its gradient, so the optimization easily gets stuck in local minima. The two fixes are the point cloud initialization, which starts the Gaussians near the right places, and the adaptive density control.
A fixed set of Gaussians cannot fit both large uniform areas and fine detail. So every 100 iterations (from iteration 500 to 15,000), the optimizer looks at Gaussians whose view-space position gradient is large on average (above 0.0002), a sign that the region is not well reconstructed:
Every 3000 iterations, all opacities are also reset close to zero. The optimizer then raises the opacity again only for the Gaussians that are needed, and the rest get pruned. This removes floaters near the cameras that would otherwise keep growing.
A typical scene ends up with one to several million Gaussians.
Strengths:
Weaknesses:
The course I followed (MIT 6.S980, Machine Learning for Inverse Graphics) describes any 3D inverse graphics method by four parts: how the field is parameterized, what it models, how it is rendered (the forward map), and how it is fitted (the inference method). In those terms, the two methods differ in the parameterization and share almost everything else:
| NeRF | 3D Gaussian splatting | |
|---|---|---|
| Field parameterization | Dense: an MLP queried anywhere in space | Sparse: millions of explicit 3D Gaussians |
| Geometry and appearance | Density $\sigma(\mathbf{x})$, view-dependent color $\mathbf{c}(\mathbf{x}, \mathbf{d})$ | Per Gaussian: opacity, covariance, spherical-harmonic color |
| Forward map | Ray marching with volume rendering quadrature | Projection to 2D, tile rasterization, alpha blending |
| Inference | Per-scene gradient descent | Per-scene gradient descent plus density control |
| Rendering speed | Tens of seconds per frame (original) | Real time, often above 100 fps |
| Storage | About 5 MB of weights | Hundreds of MB to over 1 GB |
| Editing and map updates | Hard, no locality | Easy, Gaussians are explicit |
| Surface extraction | Threshold density and run marching cubes, noisy | Not direct; use variants such as 2DGS |
The 3DGS paper compared the main methods on the Mip-NeRF 360 dataset of large indoor and outdoor scenes, on one A6000 GPU:
| Method | Training time | Rendering speed | Storage | Quality (PSNR) |
|---|---|---|---|---|
| Plenoxels | 25 min | 6.8 fps | 2.1 GB | 23.1 dB |
| Instant-NGP (large) | 7.5 min | 9.4 fps | 48 MB | 25.6 dB |
| Mip-NeRF 360 | 48 h | 0.06 fps | 8.6 MB | 27.7 dB |
| 3D Gaussian splatting | 41 min | 134 fps | 734 MB | 27.2 dB |
Gaussian splatting matches the best NeRF in quality, renders more than 2000 times faster, and trains about 70 times faster, at the price of about 100 times more storage.
A robot needs two different kinds of maps:
NeRF-based SLAM came first: iMAP and NICE-SLAM track the camera against a neural field while training it, but slow rendering and global updates limit them to small scenes. Gaussian splatting has largely taken over, because its map is explicit and local. SplaTAM, Photo-SLAM and Gaussian Splatting SLAM (MonoGS) track the camera against the Gaussian map while building it, and LIV-GaussMap fuses LiDAR, IMU and camera so the LiDAR provides accurate geometry and initialization and the camera the appearance. Open problems for real deployments include:
COLMAP for poses, the reference 3DGS implementation, nerfstudio (both nerfacto and splatfacto), the gsplat CUDA library, and phone apps such as Polycam and Scaniverse that capture and train splats end to end.