Image-based rendering focuses on rendering scenes from existing captured or rasterized images, typically from a new viewpoint.
Recent research allows adding new objects, performing relighting, and other AR effects.
Lightfields aim to capture the radiance of light rays within the scene.
Light Field Networks
This is an implicit representation similar to NeRF.
However, you directly predict colors from light rays instead of performing volume rendering.
NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts radiance at different points during volume rendering.
Multi-plane Image (MPI)
Multiple perpendicular planes each with some transparency which are composited together.
Layered Depth Image (LDI)
Multiple meshes each with some transparency. Unlike MPI, these meshes are not necessarily planes but may not correspond directly to scene objects.
- One-shot 3D photography
- Casual 3D Photography
Multi-sphere Image (MSI)
Similar to MPI but using spheres.
- Matryodshka (ECCV 2020) - Renders 6-dof video from ODS videos.
Reconstruction aims to recreate the 3D scene from a set of input images.
Techniques include structure from motion, multi-view stereo.
This type of reconstruction is also studied in the field of photogrammetry.