Image-based rendering: Difference between revisions

(14 intermediate revisions by the same user not shown)

Line 1:

Image-based rendering focuses on rendering scenes from existing captured or rasterized images~~, typically~~ from a new viewpoint.

Image-based rendering focuses on rendering scenes from existing captured or rasterized images.<br>

~~Recent~~ research ~~allow~~ adding new objects, performing relighting, and other AR effects.

View synthesis focuses on rendering the scene from a new viewpoint based on the captured information.<br>

Other research focuses on adding new objects, performing relighting, stylization, and other AR effects.

==Light Fields==

==Implicit Representations==

===Light Fields===

~~Lightfields aim to~~ capture the radiance of light rays within the scene.

Light fields capture the accumulated radiance of light rays within the scene.<br>

Traditionally stored as a grid of images or videos.

==NeRF==

===Light Field Networks===

This is an implicit representation similar to NeRF.<br>

However, you directly predict colors from light rays instead of performing volume rendering.

===NeRF===

{{ main | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis }}

NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts/~~interpolates unknown light rays based on the known light rays in the~~ scene.

NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts radiance at different points during volume rendering.

;Resources

* [https://github.com/yenchenlin/awesome-NeRF yenchenlin/awesome-NeRF]

* [https://dellaert.github.io/NeRF/ NeRF explosion 2020]

==Layered Representations==

Some notable people here are [https://scholar.google.com/citations?user=Db4BCX8AAAAJ&hl=en&oi=ao Noah Snavely] and [https://scholar.google.com/citations?user=IkpNZAoAAAAJ&hl=en&oi=sra Richard Tucker].

Representations here vary from implicit (MPI, MSI) to explicit (LDI, Point Clouds).

===Multi-plane Image (MPI)===

Multiple perpendicular planes each with some transparency which are composited together.

* [https://arxiv.org/abs/1805.09817 Stereo Magnification (SIGGRAPH 2018)]

* [https://openaccess.thecvf.com/content_CVPR_2019/html/Flynn_DeepView_View_Synthesis_With_Learned_Gradient_Descent_CVPR_2019_paper.html DeepView (CVPR 2019)]

===Layered Depth Image (LDI)===

Multiple meshes each with some transparency. Unlike MPI, these meshes are not necessarily planes but may not correspond directly to scene objects.

* [https://facebookresearch.github.io/one_shot_3d_photography/ One-shot 3D photography]

* Casual 3D Photography

===Multi-sphere Image (MSI)===

Similar to MPI but using spheres.

* [http://visual.cs.brown.edu/projects/matryodshka-webpage/ Matryodshka (ECCV 2020)] - Renders 6-dof video from ODS videos.

===Point Clouds===

* [https://www.robots.ox.ac.uk/~ow/synsin.html SynSin]

==Reconstruction==

==Classical Reconstruction==

Reconstruction aims to recreate the 3D scene from a set of input images.

Reconstruction aims to recreate the 3D scene from a set of input images, typically as a mesh or point cloud

Techniques include structure from motion, multi-view stereo.

This is also ~~known as~~ photogrammetry.

This type of reconstruction is also studied in the field of photogrammetry.

@@ Line 1: / Line 1: @@
-Image-based rendering focuses on rendering scenes from existing captured or rasterized images, typically from a new viewpoint.
+Image-based rendering focuses on rendering scenes from existing captured or rasterized images.<br>
-Recent research allow adding new objects, performing relighting, and other AR effects.
+View synthesis focuses on rendering the scene from a new viewpoint based on the captured information.<br>
+Other research focuses on adding new objects, performing relighting, stylization, and other AR effects.
-==Light Fields==
+==Implicit Representations==
+===Light Fields===
 {{ main | Light field}}
-Lightfields aim to capture the radiance of light rays within the scene.
+Light fields capture the accumulated radiance of light rays within the scene.<br>
+Traditionally stored as a grid of images or videos.
-==NeRF==
+===Light Field Networks===
+This is an implicit representation similar to NeRF.<br>
+However, you directly predict colors from light rays instead of performing volume rendering.
+===NeRF===
 {{ main | NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis }}
-NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts/interpolates unknown light rays based on the known light rays in the scene.
+NeRF preprocesses unstructured light fields into a neural network (MLP) representation which predicts radiance at different points during volume rendering.
+;Resources
+* [https://github.com/yenchenlin/awesome-NeRF yenchenlin/awesome-NeRF]
+* [https://dellaert.github.io/NeRF/ NeRF explosion 2020]
+==Layered Representations==
+Some notable people here are [https://scholar.google.com/citations?user=Db4BCX8AAAAJ&hl=en&oi=ao Noah Snavely] and [https://scholar.google.com/citations?user=IkpNZAoAAAAJ&hl=en&oi=sra Richard Tucker].
+Representations here vary from implicit (MPI, MSI) to explicit (LDI, Point Clouds).
+===Multi-plane Image (MPI)===
+Multiple perpendicular planes each with some transparency which are composited together.
+* [https://arxiv.org/abs/1805.09817 Stereo Magnification (SIGGRAPH 2018)]
+* [https://openaccess.thecvf.com/content_CVPR_2019/html/Flynn_DeepView_View_Synthesis_With_Learned_Gradient_Descent_CVPR_2019_paper.html DeepView (CVPR 2019)]
+===Layered Depth Image (LDI)===
+Multiple meshes each with some transparency. Unlike MPI, these meshes are not necessarily planes but may not correspond directly to scene objects.
+* [https://facebookresearch.github.io/one_shot_3d_photography/ One-shot 3D photography]
+* Casual 3D Photography
+===Multi-sphere Image (MSI)===
+Similar to MPI but using spheres.
+* [http://visual.cs.brown.edu/projects/matryodshka-webpage/ Matryodshka (ECCV 2020)] - Renders 6-dof video from ODS videos.
+===Point Clouds===
+* [https://www.robots.ox.ac.uk/~ow/synsin.html SynSin]
-==Reconstruction==
+==Classical Reconstruction==
-Reconstruction aims to recreate the 3D scene from a set of input images.
+Reconstruction aims to recreate the 3D scene from a set of input images, typically as a mesh or point cloud
 Techniques include structure from motion, multi-view stereo.
-This is also known as photogrammetry.
+This type of reconstruction is also studied in the field of photogrammetry.