Image-based rendering: Difference between revisions
Created page with "Image-based rendering focuses on rendering scenes from existing captured or rasterized images, typically from a new view-point but also potentially with new objects, lighting,..." |
No edit summary |
||
| (16 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
Image-based rendering focuses on rendering scenes from existing captured or rasterized images | Image-based rendering focuses on rendering scenes from existing captured or rasterized images.<br> | ||
View synthesis focuses on rendering the scene from a new viewpoint based on the captured information.<br> | |||
Other research focuses on adding new objects, performing relighting, stylization, and other AR effects. | |||
==Light Fields== | ==Implicit Representations== | ||
===Light Fields=== | |||
{{ main | Light field}} | {{ main | Light field}} | ||
Light fields capture the accumulated radiance of light rays within the scene.<br> | |||
Traditionally stored as a grid of images or videos. | |||
==NeRF== | ===Light Field Networks=== | ||
This is an implicit representation similar to NeRF.<br> | |||
However, you directly predict colors from light rays instead of performing volume rendering. | |||
===NeRF=== | |||
{{ main | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis }} | {{ main | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis }} | ||
NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts/ | NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts radiance at different points during volume rendering. | ||
;Resources | |||
* [https://github.com/yenchenlin/awesome-NeRF yenchenlin/awesome-NeRF] | |||
* [https://dellaert.github.io/NeRF/ NeRF explosion 2020] | |||
==Layered Representations== | |||
Some notable people here are [https://scholar.google.com/citations?user=Db4BCX8AAAAJ&hl=en&oi=ao Noah Snavely] and [https://scholar.google.com/citations?user=IkpNZAoAAAAJ&hl=en&oi=sra Richard Tucker]. | |||
Representations here vary from implicit (MPI, MSI) to explicit (LDI, Point Clouds). | |||
===Multi-plane Image (MPI)=== | |||
Multiple perpendicular planes each with some transparency which are composited together. | |||
* [https://arxiv.org/abs/1805.09817 Stereo Magnification (SIGGRAPH 2018)] | |||
* [https://openaccess.thecvf.com/content_CVPR_2019/html/Flynn_DeepView_View_Synthesis_With_Learned_Gradient_Descent_CVPR_2019_paper.html DeepView (CVPR 2019)] | |||
===Layered Depth Image (LDI)=== | |||
Multiple meshes each with some transparency. Unlike MPI, these meshes are not necessarily planes but may not correspond directly to scene objects. | |||
* [https://facebookresearch.github.io/one_shot_3d_photography/ One-shot 3D photography] | |||
* Casual 3D Photography | |||
===Multi-sphere Image (MSI)=== | |||
Similar to MPI but using spheres. | |||
* [http://visual.cs.brown.edu/projects/matryodshka-webpage/ Matryodshka (ECCV 2020)] - Renders 6-dof video from ODS videos. | |||
===Point Clouds=== | |||
* [https://www.robots.ox.ac.uk/~ow/synsin.html SynSin] | |||
==Reconstruction== | ==Classical Reconstruction== | ||
Reconstruction aims to recreate the 3D scene from a set of input images | Reconstruction aims to recreate the 3D scene from a set of input images, typically as a mesh or point cloud | ||
Techniques include structure from motion, multi-view stereo. | Techniques include structure from motion, multi-view stereo. | ||
This is also | This type of reconstruction is also studied in the field of photogrammetry. | ||
Latest revision as of 15:05, 12 July 2023
Image-based rendering focuses on rendering scenes from existing captured or rasterized images.
View synthesis focuses on rendering the scene from a new viewpoint based on the captured information.
Other research focuses on adding new objects, performing relighting, stylization, and other AR effects.
Implicit Representations
Light Fields
Light fields capture the accumulated radiance of light rays within the scene.
Traditionally stored as a grid of images or videos.
Light Field Networks
This is an implicit representation similar to NeRF.
However, you directly predict colors from light rays instead of performing volume rendering.
NeRF
NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts radiance at different points during volume rendering.
- Resources
Layered Representations
Some notable people here are Noah Snavely and Richard Tucker.
Representations here vary from implicit (MPI, MSI) to explicit (LDI, Point Clouds).
Multi-plane Image (MPI)
Multiple perpendicular planes each with some transparency which are composited together.
Layered Depth Image (LDI)
Multiple meshes each with some transparency. Unlike MPI, these meshes are not necessarily planes but may not correspond directly to scene objects.
- One-shot 3D photography
- Casual 3D Photography
Multi-sphere Image (MSI)
Similar to MPI but using spheres.
- Matryodshka (ECCV 2020) - Renders 6-dof video from ODS videos.
Point Clouds
Classical Reconstruction
Reconstruction aims to recreate the 3D scene from a set of input images, typically as a mesh or point cloud
Techniques include structure from motion, multi-view stereo.
This type of reconstruction is also studied in the field of photogrammetry.