Light Field Duality: Concept and Applications

Light Field Duality: Concept and Applications

Authors: George Chen, Li Hong, Kim Ng, Peter McGuinness, Christian Hofsetz, Yang Liu, Nelson Max Affiliations: STMicroelectronics, UC Davis

• ACM DL ACL DL PDF

Background

A light field is the set of all rays in a 2D space. It is a 5D function of \(\displaystyle (x,y,z)\) position and \(\displaystyle (\theta, \phi)\) angle. However for practical scenes, you can assume that the light along a ray is consistent through empty spaces (air). Thus it can be represented in 4D as either a position \(\displaystyle (u, v)\) on a fixed plane and an angle \(\displaystyle (\theta, \phi)\) or as two positions \(\displaystyle (u, v)\) and \(\displaystyle (s, t)\) on two nearby parallel planes. See Light field.

Dual Representation

The paper forms the dual representation between points in 3D world coordinates and hyperlines in the 2-plane 4D parameterization of light fields.
By hyperline, they mean a 2D manifold in 4D space that is the intersection of two 4D hyperplanes.

The duality is defined as follows:
Given a 3D point \(\displaystyle P=(x,y,z)\), any ray \(\displaystyle (s,t,u,v)\) going through P would have:
\(\displaystyle \frac{s-x}{u-x} = \frac{t-y}{v-y} = \frac{z-z_{st}}{z-z_{uv}} \) where \(\displaystyle z_{st}\), \(\displaystyle z_{uv}\) are fixed values. See the figure to the right.

With some algebra, this constraint can be written as either: \(\displaystyle \begin{bmatrix} z_{uv}-z_{st} & 0 & s-u\\ 0 & z_{uv} - z_{st} & t-v \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} sz_{uv} - uz_{st}\\ tz_{uv} - vz_{st}\\ \end{bmatrix} \)
\(\displaystyle \begin{bmatrix} z-z_{uv} & 0 & z_{st}-z & 0\\ 0 & z-z_{uv} & 0 & z_{st} - z \end{bmatrix} \begin{bmatrix}s \\ t \\ u \\ v \end{bmatrix} = \begin{bmatrix} x(z_{st}-z_{uv})\\ y(z_{st}-z_{uv}) \end{bmatrix} \)
Given a point, this constraint defines the hyperline in the 4D light field space.

Henceforth, a hyperline will be represented as \(\displaystyle (a, b, c, d)\) where hyperpoints on said line satisfy:
\(\displaystyle \begin{bmatrix} a & 0 & b & 0\\ 0 & a & 0 & b \end{bmatrix} \begin{bmatrix} s \\ t \\ u \\ v \end{bmatrix} = \begin{bmatrix} c \\ d \end{bmatrix} \)

The paper has the following remarks:

• Points in one space are lines in the other.
• A bundle of lines in world space correspond to a set of hyperlinear points in the light field.
• A set of colinear points in world space corresponds to a bundle of hyperlines in the light field.

In the light field space, you get:

• Camera hyperlines (CHL) which correspond to camera points in world space. Hyperpoints along the hyperline have heterogenous colors since they represent images.
• Geometry hyperlines (GHL) which correspond to points in the scene (on surfaces) in world space. Hyperpoints along these hyperlines have homogeneous colors assuming no view dependent effects.

CHL Light Field Rendering

Each camera \(\displaystyle i\) is a point in world coordinates which corresponds to a hyperline in the light field.
If we want to render from a new viewpoint, for each render pixel, we have a target ray which corresponds a point on the light field \(\displaystyle \mathbf{r} = (s_r, t_r, u_r, v_r)\).