Image-based rendering

From David's Wiki
\( \newcommand{\P}[]{\unicode{xB6}} \newcommand{\AA}[]{\unicode{x212B}} \newcommand{\empty}[]{\emptyset} \newcommand{\O}[]{\emptyset} \newcommand{\Alpha}[]{Α} \newcommand{\Beta}[]{Β} \newcommand{\Epsilon}[]{Ε} \newcommand{\Iota}[]{Ι} \newcommand{\Kappa}[]{Κ} \newcommand{\Rho}[]{Ρ} \newcommand{\Tau}[]{Τ} \newcommand{\Zeta}[]{Ζ} \newcommand{\Mu}[]{\unicode{x039C}} \newcommand{\Chi}[]{Χ} \newcommand{\Eta}[]{\unicode{x0397}} \newcommand{\Nu}[]{\unicode{x039D}} \newcommand{\Omicron}[]{\unicode{x039F}} \DeclareMathOperator{\sgn}{sgn} \def\oiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x222F}\,}{\unicode{x222F}}{\unicode{x222F}}{\unicode{x222F}}}\,}\nolimits} \def\oiiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x2230}\,}{\unicode{x2230}}{\unicode{x2230}}{\unicode{x2230}}}\,}\nolimits} \)

Image-based rendering focuses on rendering scenes from existing captured or rasterized images, typically from a new viewpoint.
Recent research allows adding new objects, performing relighting, and other AR effects.

Implicit Representations

Light Fields

Lightfields aim to capture the radiance of light rays within the scene.

NeRF

NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts/interpolates unknown light rays based on the known light rays in the scene.

Resources

Layered Representations

Some notable people here are Noah Snavely and Richard Tucker.
Representations here vary from implicit (MPI, MSI) to explicit (LDI, Point Clouds).

Multi-plane Image (MPI)

Layered Depth Image (LDI)

Multi-sphere Image (MSI)

Point Clouds

Classical Reconstruction

Reconstruction aims to recreate the 3D scene from a set of input images.
Techniques include structure from motion, multi-view stereo.
This is also known as photogrammetry.