3D Representations

Ways to represent 3D objects and scenes.

Representations

Collection of images

Just a set of 2D images. Videos may also fall under this category.
Images may or may not have depth and may or may not be posed.

Light Field

This is similar to a collection of images.
However, instead of thinking of it as a set of images, each pixel represents the color along a ray in 3D space.
Thus, light fields are collections of rays in 3D space which can be modeled using the plenoptic function \(\displaystyle F: \mathbb{R}^5 \to \mathbb{R}\).
3-dimensions for the starting point of the ray and 2 dimensions for the direction of the ray.
Typically we assume the rays lie in an empty/transparent space until they hit a surface thus rays can be models as lines which can be defined with 4 dimensions.
See Light field.

Volume

A volume is a dense 3D dataset \(\displaystyle F: \mathbb{R}^3 \to \mathbb{R}\) where the value at each position \(\displaystyle (x,y,z)\) corresponds to a density.
Volume rendering is used project a 3D volume into a 2D image by sampling the volume along a ray for each pixel in the 2D image.
Also known as a radiance field. These can be encoded as explicit functions or neural networks.

Signed Distance Field

A signed distance field is a dense 3D dataset \(\displaystyle F: \mathbb{R}^2 \to \mathbb{R}\) where each position represents a distance to the nearest surface.
Positive values represent a point on the exterior of an object whereas negative values represent a point on the interior.
Ray marching is used to render signed distance fields into 2D images.

Point-clouds

A list of points in 3D (\(\displaystyle N \times 3\)). Points may also have color or other values.
Typically points are not semantically labeled so points are entirely unrelated to each other.

Mesh

A list of vertices (\(\displaystyle N \times 3\)) and triangles (\(\displaystyle N \times 3\)).
The current standard representation used in real-time 3D applications.

CAD Models

These are vector representations with relationships between objects.
E.g. Object A is left of Object B by 5 centimeters.

Conversions

Volume rendering

Volume rendering converts volumes to 2D images.
Differentiable volume rendering (NeRF) can convert posed 2D images to neural volumes.

Ray marching

Ray marching converts signed distance fields or radiance fields to 2D images.

Rasterization

Rasterization converts meshes to 2D images.

Marching cubes

Marching cubes converts volumes to meshes.

Reconstruction

Structure from motion and multi view stereo methods can build point clouds from 2D images.
Meshing, or surface reconstruction, works on converting this point clouds into meshes.

See Also

• Image-based rendering