Image-based rendering

From David's Wiki
Revision as of 15:03, 12 July 2023 by David (talk | contribs)
\( \newcommand{\P}[]{\unicode{xB6}} \newcommand{\AA}[]{\unicode{x212B}} \newcommand{\empty}[]{\emptyset} \newcommand{\O}[]{\emptyset} \newcommand{\Alpha}[]{Α} \newcommand{\Beta}[]{Β} \newcommand{\Epsilon}[]{Ε} \newcommand{\Iota}[]{Ι} \newcommand{\Kappa}[]{Κ} \newcommand{\Rho}[]{Ρ} \newcommand{\Tau}[]{Τ} \newcommand{\Zeta}[]{Ζ} \newcommand{\Mu}[]{\unicode{x039C}} \newcommand{\Chi}[]{Χ} \newcommand{\Eta}[]{\unicode{x0397}} \newcommand{\Nu}[]{\unicode{x039D}} \newcommand{\Omicron}[]{\unicode{x039F}} \DeclareMathOperator{\sgn}{sgn} \def\oiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x222F}\,}{\unicode{x222F}}{\unicode{x222F}}{\unicode{x222F}}}\,}\nolimits} \def\oiiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x2230}\,}{\unicode{x2230}}{\unicode{x2230}}{\unicode{x2230}}}\,}\nolimits} \)

Image-based rendering focuses on rendering scenes at a new viewpoint from existing captured or rasterized images. Recent research allows adding new objects, performing relighting, and other AR effects.

Implicit Representations

Light Fields

Light fields capture the accumulated radiance of light rays within the scene.
Traditionally stored as a grid of images or videos.

Light Field Networks

This is an implicit representation similar to NeRF.
However, you directly predict colors from light rays instead of performing volume rendering.

NeRF

NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts radiance at different points during volume rendering.

Resources

Layered Representations

Some notable people here are Noah Snavely and Richard Tucker.
Representations here vary from implicit (MPI, MSI) to explicit (LDI, Point Clouds).

Multi-plane Image (MPI)

Multiple perpendicular planes each with some transparency which are composited together.

Layered Depth Image (LDI)

Multiple meshes each with some transparency. Unlike MPI, these meshes are not necessarily planes but may not correspond directly to scene objects.

Multi-sphere Image (MSI)

Similar to MPI but using spheres.

Point Clouds

Classical Reconstruction

Reconstruction aims to recreate the 3D scene from a set of input images, typically as a mesh or point cloud
Techniques include structure from motion, multi-view stereo.
This type of reconstruction is also studied in the field of photogrammetry.