NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis: Difference between revisions
No edit summary |
|||
| (5 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
NeRF Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) | NeRF Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) | ||
Authors: | * Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng | ||
* Affiliations: UC Berkeley, Google Research, UC San Diego | |||
* [https://www.matthewtancik.com/nerf Webpage] | * [https://www.matthewtancik.com/nerf Webpage] | ||
The main idea is to do volume rendering through an MLP.<br> | |||
As volume rendering is differentiable, the MLP will learn color and density at each position.<br> | |||
Training is supervised on multiple posed static images (e.g. 25 images). Poses can be obtained using COLMAP. | |||
Successful execution relies on two tricks: | |||
* A positional encoding. | |||
* Hierarchical volume sampling | |||
==Method== | ==Method== | ||
===Volume Rendering=== | |||
===Positional Encoding=== | |||
This is further explored in a followup paper<ref name="tancik2020fourier"></ref>. | |||
===Hierarchical volume sampling=== | |||
This idea is to uses a course network to influence sampling of the fine network. | |||
==Architecture== | |||
They use a standard MLP with extra inputs.<br> | |||
Their MLP has 9 layers with 256-dim features and one layer with 128-dim features (~600k parameters?).<br> | |||
See their paper and code for more details. | |||
==Experiments== | ==Experiments== | ||
They compare with their past work on Local Light Field Fusion as well as a few other view synthesis papers. | |||
They also publish a dataset available [https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1 on their Google Drive]. | |||
==Followup Work== | ==Followup Work== | ||
See [https://github.com/yenchenlin/awesome-NeRF yenchenlin/awesome-NeRF] | See [https://github.com/yenchenlin/awesome-NeRF yenchenlin/awesome-NeRF] | ||
===Positional Encoding=== | |||
* [https://arxiv.org/abs/2006.10739 Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains] | |||
===Videos=== | |||
* [https://arxiv.org/abs/2011.12948 Deformable Neural Radiance Fields] | |||
* [https://arxiv.org/pdf/2011.12950.pdfSpace-time Neural Irradiance Fields for Free-Viewpoint Video] | |||
* [https://neural-3d-video.github.io/ Neural 3D Video Synthesis from Multi-view Video (CVPR 2022] | |||
==References== | |||
{{reflist|refs= | |||
<ref name="tancik2020fourier">Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., ... & Ng, R. (2020). Fourier features let networks learn high frequency functions in low dimensional domains. [https://arxiv.org/abs/2006.10739 Arxiv]</ref> | |||
}} | |||
[[Category: Papers]] | |||
Latest revision as of 16:42, 17 April 2023
NeRF Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020)
- Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
- Affiliations: UC Berkeley, Google Research, UC San Diego
The main idea is to do volume rendering through an MLP.
As volume rendering is differentiable, the MLP will learn color and density at each position.
Training is supervised on multiple posed static images (e.g. 25 images). Poses can be obtained using COLMAP.
Successful execution relies on two tricks:
- A positional encoding.
- Hierarchical volume sampling
Method
Volume Rendering
Positional Encoding
This is further explored in a followup paper[1].
Hierarchical volume sampling
This idea is to uses a course network to influence sampling of the fine network.
Architecture
They use a standard MLP with extra inputs.
Their MLP has 9 layers with 256-dim features and one layer with 128-dim features (~600k parameters?).
See their paper and code for more details.
Experiments
They compare with their past work on Local Light Field Fusion as well as a few other view synthesis papers.
They also publish a dataset available on their Google Drive.
Followup Work
Positional Encoding
Videos
- Deformable Neural Radiance Fields
- Neural Irradiance Fields for Free-Viewpoint Video
- Neural 3D Video Synthesis from Multi-view Video (CVPR 2022