5,322
edits
Unsupervised Learning: Difference between revisions
→Generative Models
(→VAEs) |
|||
| Line 174: | Line 174: | ||
==Generative Models== | ==Generative Models== | ||
Goal: Generate realistic but fake samples. | |||
Applications: Denoising, impainting | |||
===VAEs=== | ===VAEs=== | ||
Variational Auto-Encoders<br> | Variational Auto-Encoders<br> | ||
[https://arxiv.org/abs/1606.05908 Tutorial]<br> | [https://arxiv.org/abs/1606.05908 Tutorial]<br> | ||
====KL Divergence==== | ====KL Divergence==== | ||
[[Wikipedia:Kullback–Leibler divergence]]<br> | * [[Wikipedia:Kullback–Leibler divergence]]<br> | ||
Kullback–Leibler divergence<br> | * Kullback–Leibler divergence<br> | ||
<math>KL(P \Vert Q) =E_{P}\left[ \log(\frac{P(X)}{Q(X)} \right]</math> | * <math>KL(P \Vert Q) =E_{P}\left[ \log(\frac{P(X)}{Q(X)}) \right]</math> | ||
; Notes | |||
* KL is always >= 0 | |||
* KL is not symmetric | |||
====Model==== | |||
Our model for how the data is generated is as follows: | |||
* Generate latent variables <math>z^{(1)},...,z^{(m)} \in \mathbb{R}^r<math> iid where dimension r is less than n. | |||
** We assume <math>Z^{(i)} \sim N(\mathbf{0},\mathbf{I})</math> | |||
* Generate <math>x^{(i)}</math> where <math>X^{(i)} \vert Z^{(i)} \sin N(g_{\theta}(z), \sigma^2 \mathbf{I})</math> | |||
** For some function <math>g_{\theta_1}}</math> parameterized by <math>\theta_1</math> | |||
====Variational Bound==== | |||
The variational bound is: | |||
* <math>\log P(x^{(i)}) \geq E_{Z}[\log P(X^{(i) \vert Z)] - KL(Q_i(z) \Vert P(z))</math> | |||
{{hidden | Derivation | | |||
We know from Baye's rule | |||
}} | |||