5,337
edits
Deep Learning: Difference between revisions
→Flow-based Generative Models
| Line 1,419: | Line 1,419: | ||
An <math>f</math> function which is element-wise would have a diagonal jacobian. This is not very expressive. | An <math>f</math> function which is element-wise would have a diagonal jacobian. This is not very expressive. | ||
===Upper-triangular Jacobian=== | |||
RealNVP by [Dint et al] considers an upper-triangular matrix. | RealNVP by [Dint et al] considers an upper-triangular matrix. | ||
In this case, the determinant is still the diagonal. | In this case, the determinant is still the diagonal. | ||
| Line 1,435: | Line 1,436: | ||
\end{pmatrix} | \end{pmatrix} | ||
</math> | </math> | ||
and <math>det(J) = \prod (S_\theta)_i</math>. | and <math>det(J) = \prod (S_\theta)_i</math>. | ||
Is this expressive enough? No. | |||
===Composition of transformations=== | |||
<math>f = f_1 \circ f_2 \circ ... \circ f_k</math> | |||
A sequence of invertible transformations is called normalizing flows. | |||
Here, we have the following properties | |||
* <math>(f_1 \circ f_2)^{-1} = f_2^{-1} \circ f_1^{-1}</math> | |||
* <math>\nabla(f_2 \circ f_1)(x) = \nabla f_2(f_1(x)) \nabla f_1(x)</math> | |||
* <math>det(J_1 J_2) = det(J_1) det(J_2)</math> | |||
RealNVP uses two types of partitioning | |||
* Checkerboard | |||
* Channel partitioning | |||
In Glow [Kingmen et al], they use invertible 1x1 convolution which is equivalent to matrix multiplication. | |||
Use <math>W = P L (U+diag(s))</math> and <math>| det(W)| = sum(diag |S|)</math> | |||
==Misc== | ==Misc== | ||