View source for Space-time Neural Irradiance Fields for Free-Viewpoint Video

Space-time Neural Irradiance Fields for Free-Viewpoint Video

Authors: Wenqi Xian, Jia-Bin Huang, Johannes Kopf, Changil Kim
Affiliations: Cornell, Virginia Tech, Facebook

* [https://video-nerf.github.io/ Website]
* [https://arxiv.org/abs/2011.12950 Paper]

The main contribution of this paper is the idea to use depth (either ground-truth or estimated) as an additional means of supervision, addressing motion-appearance ambiguity when storing videos in NeRF.

==Method==
* Inputs: Stream of RGBD images (i.e. an RGBD video) with camera matrices.
* Model: F maps a spatial-temporal location <math>(\mathbf{x}, t)</math> to a color <math>\mathbf{c}</math> and density <math>\sigma</math>. <math>F: (\mathbf{x}, t) \to (\mathbf{c}, \sigma)</math>

===Losses===
;L2 RGB Loss
<math>\sum \Vert \hat{C}(\mathbf{r}, t) - C(\mathbf{r}, t) \Vert_2^2</math>

;Depth reconstruction loss
<math>\sum \Vert \frac{1}{\hat{D}(\mathbf{r}, t)} - \frac{1}{D(\mathbf{r}, t)} \Vert_2^2</math><br>
To estimate depth from volume rendering, they use <math>\hat{D}(\mathbf{r}, t) = \int_{s_n}^{s_f}T(s,t) \sigma(\mathbf{r}(s),t)ds</math><br>
where T is the accumulated transmittance up to the point s: <math>T(s) = \exp \left( - \int_{s_n}^{s} \sigma(\mathbf{r}(p))dp \right)</math>

;Empty space loss
<math>L_{empty} = \sum \int_{s_n}^{d_t(u) - \epsilon} \sigma(\mathbf{r}(s), t) ds</math>.<br>
This loss penalizes density before ray reaches the actual depth.<br>
This is used to ensure the volume isn't too sparse since the depth loss only considers the mean of samples along the ray not the variance.

;Static scene loss
The goal is to ensure that occluded regions do not yield artifacts when viewing from a new perspective.  
To achieve this, they just force the occluded regions, i.e. those beyond the ground-truth depth, to stay static over time.<br>
<math>L_{static} = \sum \Vert F(\mathbf{x}, t) - F(\mathbf{x}, t') \Vert_2^2</math>

==Architecture==
Architecture is the same as the original NeRF.

===Training===
During training, they use <math>N_r=1024</math> rays randomly drawn w/o replacement for each iteration.<br>
Rays are sampled uniformly in inverse depth.<br>
Takes 50 hours to train with 100 frames at <math>960 \times 540</math> on two V100s.

==Evaluation==

==References==
{{reflist|refs=
}}
[[Category: Papers]]