Neural Fields: Difference between revisions
| Line 58: | Line 58: | ||
The idea here is to have an MLP which models the deformation of a canonical frame to the target frame. | The idea here is to have an MLP which models the deformation of a canonical frame to the target frame. | ||
* [https://nerfies.github.io/ Nerfies: Deformable Neural Radiance Fields] | |||
** Windowed positional encoding | |||
* [https://www.albertpumarola.com/research/D-NeRF/index.html D-NeRF (CVPR 2020)] | * [https://www.albertpumarola.com/research/D-NeRF/index.html D-NeRF (CVPR 2020)] | ||
* [https://hypernerf.github.io/ HyperNeRF (SIGGRAPH Asia 2021)] | * [https://hypernerf.github.io/ HyperNeRF (SIGGRAPH Asia 2021)] | ||
** Allows the deformation network to output 2 additional feature values which ''slice'' the canonical NeRF. | |||
====Latent code==== | ====Latent code==== | ||
| Line 68: | Line 71: | ||
====Time-axis==== | ====Time-axis==== | ||
* [https://video-nerf.github.io/ Space-time Neural Irradiance Fields for Free-Viewpoint Video (Video-NeRF)] | |||
** Adds a bunch of regularization which allows directly inputting time to the MLP. | |||
* [https://aoliao12138.github.io/FPO/ Fourier PlenOctrees] | |||
** Apply DFT to spherical harmonics in PlenOctrees. | |||
* [https://arxiv.org/pdf/2202.06088.pdf NeuVV] | |||
** Hyperspherical Harmonics | |||
* [https://arxiv.org/abs/2301.11280 Text-To-4D Dynamic Scene Generation (2023)] | * [https://arxiv.org/abs/2301.11280 Text-To-4D Dynamic Scene Generation (2023)] | ||
** Extends the tri-plane feature grid to a six-plane feature grid ({x, y, z, t} choose 2). | ** Extends the tri-plane feature grid to a six-plane feature grid ({x, y, z, t} choose 2). | ||
| Line 74: | Line 83: | ||
Segment static background and objects from dynamic background and objects | Segment static background and objects from dynamic background and objects | ||
* [https://arxiv.org/pdf/2104.14786.pdf Editable Free-Viewpoint Video using a Layered Neural Representation] | |||
** Create a scene by compositing one NeRF per actor. | |||
* [https://arxiv.org/abs/2303.03361 NeRFlets (2023)] | * [https://arxiv.org/abs/2303.03361 NeRFlets (2023)] | ||
* [https://arxiv.org/abs/2303.14536 SUDS: Scalable Urban Dynamic Scenes (2023)] | * [https://arxiv.org/abs/2303.14536 SUDS: Scalable Urban Dynamic Scenes (2023)] | ||
Revision as of 15:23, 30 March 2023
Neural Fields refers to using neural networks or neural methods to represent scenes or other signals in computer vision and graphics.
Techniques
Forward Maps
Forward maps are the differentiable functions which convert the representation to an observed signal.
Shapes
- Occupancy Grids or Voxel Grids
- Signed Distance Functions
- Primary-ray (PRIF)
3D Scenes
- Radiance Fields (NeRF)
- Light Fields
Identity
- Images
Architectures
Neural Networks
- MLP
- CNN + MLP
- Progressive Architectures
Hybrid Representations
- Voxel Grids
These typically combine a octree or voxel grid with an MLP.
Some of these are basically feature grids.
- Neural Sparse Voxel Fields
- KiloNeRF
- Point Clouds
- Mesh
Feature Grids
- Plenoxels
- Plenoctrees
- Hash (Instant-NGP)
- Vector Quantization
https://nv-tlabs.github.io/vqad/
- Factorized Feature Grids
- TensoRF
Dynamic Content
Deformation
The idea here is to have an MLP which models the deformation of a canonical frame to the target frame.
- Nerfies: Deformable Neural Radiance Fields
- Windowed positional encoding
- D-NeRF (CVPR 2020)
- HyperNeRF (SIGGRAPH Asia 2021)
- Allows the deformation network to output 2 additional feature values which slice the canonical NeRF.
Latent code
- Neural 3D Video Synthesis (CVPR 2022)
- Use a latent code for each time step.
- NeRFPlayer
- Use a time-dependent sliding window along the feature channels in a feature grid.
Time-axis
- Space-time Neural Irradiance Fields for Free-Viewpoint Video (Video-NeRF)
- Adds a bunch of regularization which allows directly inputting time to the MLP.
- Fourier PlenOctrees
- Apply DFT to spherical harmonics in PlenOctrees.
- NeuVV
- Hyperspherical Harmonics
- Text-To-4D Dynamic Scene Generation (2023)
- Extends the tri-plane feature grid to a six-plane feature grid ({x, y, z, t} choose 2).
Segmentation
Segment static background and objects from dynamic background and objects
- Editable Free-Viewpoint Video using a Layered Neural Representation
- Create a scene by compositing one NeRF per actor.
- NeRFlets (2023)
- SUDS: Scalable Urban Dynamic Scenes (2023)
Generalization
Generalization mainly focuses on learning a prior over the distribution, similar to what existing image generation network do.
This enables more advanced vision tasks such as view synthesis from a single image, shape completion, inpainting, object generation, segmentation.
- CNN
- pixelNeRF
- Latent Codes
- Hyper Networks
- Light Field Networks
Applications
3D Generation
- EG3D - Adapting Stylegan2, NeRF, and a super-resolution network for generating 3D scenes
- Dream Fields - CLIP-guided NeRF generation
- Dreamfusion - Adapting text-to-image diffusion models to generate NeRFs