\( \newcommand{\P}[]{\unicode{xB6}} \newcommand{\AA}[]{\unicode{x212B}} \newcommand{\empty}[]{\emptyset} \newcommand{\O}[]{\emptyset} \newcommand{\Alpha}[]{Α} \newcommand{\Beta}[]{Β} \newcommand{\Epsilon}[]{Ε} \newcommand{\Iota}[]{Ι} \newcommand{\Kappa}[]{Κ} \newcommand{\Rho}[]{Ρ} \newcommand{\Tau}[]{Τ} \newcommand{\Zeta}[]{Ζ} \newcommand{\Mu}[]{\unicode{x039C}} \newcommand{\Chi}[]{Χ} \newcommand{\Eta}[]{\unicode{x0397}} \newcommand{\Nu}[]{\unicode{x039D}} \newcommand{\Omicron}[]{\unicode{x039F}} \DeclareMathOperator{\sgn}{sgn} \def\oiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x222F}\,}{\unicode{x222F}}{\unicode{x222F}}{\unicode{x222F}}}\,}\nolimits} \def\oiiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x2230}\,}{\unicode{x2230}}{\unicode{x2230}}{\unicode{x2230}}}\,}\nolimits} \)

Neural Fields refers to using neural networks or neural methods to represent scenes or other signals in computer vision and graphics.

Techniques

Forward Maps

Forward maps are the differentiable functions which convert the representation to an observed signal.

Shapes

Occupancy Grids or Voxel Grids
Signed Distance Functions
Primary-ray (PRIF)

3D Scenes

Radiance Fields (NeRF)
Light Fields

Identity

Images

Architectures

Neural Networks

MLP
  • SIREN
    • Proposes using sine activation to remove the spectrial bias instead of positional encoding.
  • SAPE
    • Progressively exposes additional frequencies during training based on time and space.
CNN + MLP
Progressive Architectures

Hybrid Representations

Voxel Grids

These typically combine a octree or voxel grid with an MLP.
Some of these are basically feature grids.

  • Neural Sparse Voxel Fields
  • KiloNeRF
    • Uses thousands of small voxels, each modelled by a single NeRF. Optimized using a teacher network.
Point Clouds
Mesh

Feature Grids

  • Plenoctrees
    • Convert a NeRF into a octree of spherical harmonics for fast rendering.
  • Plenoxels
    • Directly use a voxel grid of spherical harmonics to fast optimization and rendering.
  • Hash (Instant-NGP)
    • Use a hash function map voxels to features in a codebook. Disconnects grid resolution from codebook size.
  • Variable Bitrate Neural Fields
    • Use vector quantization to compress feature grids. However, need to store an grid of indices.
Factorized Feature Grids
  • TensoRF

Dynamic Content

Deformation

The idea here is to have an MLP which models the deformation of a canonical frame to the target frame.

Latent code

Time-axis

Segmentation

Segment static background and objects from dynamic background and objects

Generalization

Generalization mainly focuses on learning a prior over the distribution, similar to what existing image generation network do.
This enables more advanced vision tasks such as view synthesis from a single image, shape completion, inpainting, object generation, segmentation.

CNN
  • pixelNeRF
Latent Codes
Hyper Networks
  • Light Field Networks

Applications

3D Generation

  • EG3D - Adapting Stylegan2, NeRF, and a super-resolution network for generating 3D scenes
  • Dream Fields - CLIP-guided NeRF generation
  • Dreamfusion - Adapting text-to-image diffusion models to generate NeRFs


Resources