Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering

Authors:Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, Fredo Durand
Affiliations: MIT Links https://arxiv.org/abs/2106.02634

Method

Background

See NeRF and SIREN.

Light Field Networks

The idea here is to use light field rendering instead of volume rendering or SDF ray marching.
In this case, the input to the network is the entire ray rather than the a single 3D point.
Thus, it is not necessary to sample across the entire ray and composite the samples.

Plucker coordinates

They use Plucker coordinates to encode rays instead of directly inputting the (point, direction) representation or using a two-plane parameterization.
The benefit is that Plucker coordinates are invariance to the selected point and can represent the entire 360 set of rays.
\(\displaystyle \mathbf{r} = (\mathbf{d},\mathbf{m}) \in \mathbb{R}^6\) where \(\displaystyle \mathbf{m}=\mathbf{p} \times \mathbf{d}\)

Geometry

(NOT FILLED IN)
There is some interesting discussion in the paper about the point-line isomorphism, epipolar plane image, and how to extract depth.

Metalearning

They use a hypernetwork to convert latent codes to scenes represented by the networks.

Evaluation