NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis: Difference between revisions
(→Method) |
No edit summary |
||
Line 1: | Line 1: | ||
NeRF Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) | NeRF Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) | ||
Authors: | * Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng | ||
* Affiliations: UC Berkeley, Google Research, UC San Diego | |||
* [https://www.matthewtancik.com/nerf Webpage] | * [https://www.matthewtancik.com/nerf Webpage] | ||
The main idea is to do volume rendering through an MLP.<br> | |||
As volume rendering is differentiable, the MLP will learn color and density at each position.<br> | |||
Training is supervised on multiple posed static images (e.g. 25 images). Poses can be obtained using COLMAP. | |||
Successful execution relies on two tricks: | |||
* A positional encoding. | |||
* Hierarchical volume sampling | |||
==Method== | ==Method== | ||
===Volume Rendering=== | |||
===Positional Encoding=== | |||
This is further explored in a followup paper<ref name="tancik2020fourier"></ref>. | |||
===Hierarchical volume sampling=== | |||
This idea is to uses a course network to influence sampling of the fine network. | |||
==Architecture== | ==Architecture== | ||
They use a standard MLP with extra inputs.<br> | |||
Their MLP has 9 layers with 256-dim features and one layer with 128-dim features (~600k parameters?).<br> | |||
See their paper and code for more details. | |||
==Experiments== | ==Experiments== | ||
They compare with their past work on Local Light Field Fusion as well as a few other view synthesis papers. | |||
They also publish a dataset available [https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1 on their Google Drive]. | |||
==Followup Work== | ==Followup Work== | ||
See [https://github.com/yenchenlin/awesome-NeRF yenchenlin/awesome-NeRF] | See [https://github.com/yenchenlin/awesome-NeRF yenchenlin/awesome-NeRF] | ||
===Positional Encoding=== | |||
* [https://arxiv.org/abs/2006.10739 Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains] | |||
===Videos=== | ===Videos=== | ||
* [https://arxiv.org/abs/2011.12948 Deformable Neural Radiance Fields] | * [https://arxiv.org/abs/2011.12948 Deformable Neural Radiance Fields] | ||
* [https://arxiv.org/pdf/2011.12950.pdfSpace-time Neural Irradiance Fields for Free-Viewpoint Video] | * [https://arxiv.org/pdf/2011.12950.pdfSpace-time Neural Irradiance Fields for Free-Viewpoint Video] | ||
==References== | |||
{{reflist| | |||
<ref name="tancik2020fourier">Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., ... & Ng, R. (2020). Fourier features let networks learn high frequency functions in low dimensional domains. [https://arxiv.org/abs/2006.10739 Arxiv]</ref> | |||
}} |
Revision as of 21:44, 8 February 2021
NeRF Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020)
- Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
- Affiliations: UC Berkeley, Google Research, UC San Diego
The main idea is to do volume rendering through an MLP.
As volume rendering is differentiable, the MLP will learn color and density at each position.
Training is supervised on multiple posed static images (e.g. 25 images). Poses can be obtained using COLMAP.
Successful execution relies on two tricks:
- A positional encoding.
- Hierarchical volume sampling
Method
Volume Rendering
Positional Encoding
This is further explored in a followup paper[1].
Hierarchical volume sampling
This idea is to uses a course network to influence sampling of the fine network.
Architecture
They use a standard MLP with extra inputs.
Their MLP has 9 layers with 256-dim features and one layer with 128-dim features (~600k parameters?).
See their paper and code for more details.
Experiments
They compare with their past work on Local Light Field Fusion as well as a few other view synthesis papers.
They also publish a dataset available on their Google Drive.