View source for SfM-Net: Learning of Structure and Motion from Video

SfM-Net: Learning of Structure and Motion from Video (2017)

Authors: Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, Katerina Fragkiadaki
Affiliations: Google, Inria Research Institute, CMU
* [https://arxiv.org/abs/1704.07804 Arxiv]

SfM-Net is a geometry-aware neural network for motion estimation in videos.  
From two video frames, the CNN can regress scene depth, camera rotation+translation, motion masks, and 3D rigid rotations and translations.  
These can be backprojected into 3D scene flow and 2D optical flow for frame interpolation.

==Method==
The first build two networks to estimate the following for time \(t\):
* Image depth \(d_t \in [0,\infty)^{w \times h} \).
* Camera rotation and translation: \(\{R_t^c, t_t^c \}\)
* K motion masks: \(m_t^k\) for \(k=1,...,K\)
* K object motions: \(\{R_t^k, t_t^k \}\)

The data above is used as follows:
* Use the depth to generate a point cloud
* Transform the point cloud based on object transformations
* Transform the point cloud based on camera transformations
* Compute optical flow and do warping
* Repeat with \(I_{t+1}\) and \(I_{t}\) for backward consistency

===Supervision/Loss Function===
They apply several forms of supervision:

* Self-supervision: Minimize distance between the reference and the warped image
* Spatial smoothness priors: Penalize l1 norm of gradients on optical flow field, depth, and motion maps
* Forward-backward consistency constraints: Do a run backwards in time and make sure the depth \(d_{t+1}\) is consistent with \(d_t\)
* Supervising depth: Minimize estimate and ground truth depth
* Supervising camera motion: Minimize estimate and ground truth camera motion
* Supervising optical flow and object motion: Minimize estimate and ground truth optical flow and object motion on synthetic datasets

==Architecture==
SfM-Net consists of two neural networks:  
* The motion network estimates camera motion, object motion, and object masks.
* The structure network estimates depth what can be used to make a point cloud.

Both networks follow a Conv-Deconv (U-Net) structure with skip connections.  
See the figure in the paper for more details

===Motion Network===
The inputs to the motion network are a pair of video frames \(I_t\) and \(I_{t+1}\) totaling a tensor with shape (\(380 \times 128 \times 6\).  
From this, the motion network predicts the following:
* Camera rotation and translation: \(\{R_t^c, t_t^c \}\)
* K motion masks: \(m_t^k\) for \(k=1,...,K\)
* K object motions: \(\{R_t^k, t_t^k \}\)


===Structure Network===
The goal of the structure network is to estimate depth \(d_t \in [0,\infty)^{w \times h} \).  


==Evaluation==
The do an evaluation on the KITTI 2012 & 2015 datasets, MoSeg dataset, and RGB-D SLAM datasets.  
They only provide comparisons to ground-truth and not other approaches.

==References==

[[Category: Papers]]