View source for SinGAN: Learning a Generative Model from a Single Natural Image

SinGAN<br>
[https://arxiv.org/abs/1905.01164 Paper]<br>
[http://webee.technion.ac.il/people/tomermic/SinGAN/SinGAN.htm Author's Website]<br>
[https://tomer.net.technion.ac.il/files/2019/09/SingleImageGan_SM.pdf Supplementary Material]
[http://openaccess.thecvf.com/content_ICCV_2019/supplemental/Shaham_SinGAN_Learning_a_ICCV_2019_supplemental.pdf Mirror(CVF Host)]<br>
[http://openaccess.thecvf.com/content_ICCV_2019/html/Shaham_SinGAN_Learning_a_Generative_Model_From_a_Single_Natural_Image_ICCV_2019_paper.html ICCV]<br>
[https://github.com/tamarott/SinGAN Github Official PyTorch Implementation]<br>
SinGAN: Learning a Generative Model from a Single Natural Image<br>
Authors: 
[https://stamarot.webgr.technion.ac.il/ Tamar Rott Shaham] ([https://www.technion.ac.il/ Technion]), 
[http://people.csail.mit.edu/talidekel/ Tali Dekel] (Google Research), 
[http://webee.technion.ac.il/people/tomermic/ Tomer Michaeli] (Technion)<br>

[[File: Singan_teaser.PNG | 800x400px]]


==Basic Idea==
<!--
This is wrong there are no patches in the code.
Bootstrap patches of the original image and build GANs which can add fine details to blurry patches at different path sizes.
-->
Train GANs to fill in details at different scales of the image
* Start by building a GAN to generate low-resolution versions of the original image
* Then upscale the image and build a GAN to add details to your upscaled image
* Fix the parameters of the previous GAN. Upscale the outputs and repeat.

==Architecture==
They build <math>N</math> PatchGANs, N usually 7-8<br>
Each GAN <math>G_n</math> adds details to the image produced by GAN <math>G_{n+1}</math> below it.<br>
The final GAN <math>G_0</math> adds only fine details.

===Generator===
The use N generators which they call a hierarchy of patch-GANs.<br>
Each generator consists of 5 convolutional blocks:<br>
Conv(<math>3 \times 3</math>)-BatchNorm-LeakyReLU.<br>
Note: This generator is similar to pix2pix.<br>
They use 32 kernels per block at the coarsest scale and increase <math>2 \times</math> every 4 scales.<br>
This means their convolutional layers have an input and output of 32 channels.

====[https://arxiv.org/pdf/1502.03167.pdf Batch Normalization]====
; Definitions:
* Internal Covariate Shift - "the change in distribution of network activations as network parameters change"
* Whitening - "linearly transformed to have zero means and unit variances, and decorrelated"
The idea is to do a element-wise normalization per minibatch to improve training speed.
<pre>
B <- get minibatch
u_B <- calculate minimatch mean
sigma2_B <- calculate minibatch variance
x_hat <- (x - u_B) / sqrt(sigma2_B + eps) // normalize
y <- gamma*x + b // scale and shift
</pre>
* <math>\epsilon</math> is added for numerical stability (e.g. if sigma is small)
* In the algorithm, <math>\gamma</math> and <math>\beta</math> are learned parameters.
* The final scale and shift allow the entire BN process to be an identity transformation
** The BN paper gives a sigmoid activation as an example which is linear around 0.
:: Batch Normalization would almost eliminate the non-linearity with sigmoid

====Leaky Relu====
Relu is 
<math>\begin{cases}
x & \text{if }x > 0\\
0 & \text{if }x <= 0
\end{cases}
</math>. <br>
If the input is <math><=0</math> then any gradient through that neuron will always be 0.<br>
This leads to dead neurons which remain dead if the neurons below never output a positive number.<br>
That is, you get neurons which always output <math>0</math> throughout the training process.<br>
Leaky relu: <math>
\begin{cases}
x & \text{if }x > 0\\
0.01x & \text{if }x <= 0
\end{cases}
</math>
always has a gradient so neurons below will always be updated.<br>
SinGAN uses a negative slope of 0.2 (instead of the default 0.01).

===Discriminator===
The architecture is the same as the generator.<br>
The patch size is <math>11 \times 11</math>
The GAN used is PatchGAN from [https://arxiv.org/abs/1611.07004 pix2pix].<br>
PatchGAN's discriminator is referred to as a Markovian discriminator because the receptive field is smaller than the size of the image.<br>
"Such a discriminator effectively models the image as a Markov random field, assuming independence between pixels separated by more than a patch diameter."

==Training and Loss Function==
<math>\min_{G_n} \max_{D_n} \mathcal{L}_{adv}(G_n, D_n) + \alpha \mathcal{L}_{rec}(G_n)</math><br>
They use a combination of the standard GAN adversarial loss and a reconstruction loss.

===Adversarial Loss===
They use the [https://arxiv.org/abs/1704.00028 WGAN-GP loss].<br>
This drops the log from the traditional cross-entropy loss.<br>
<math>\min_{G_n}\max_{D_n}L_{adv}(G_n, D_n)+\alpha L_{rec}(G_n)</math><br>
<!--
The final loss is the average over all the patches.<br>
-->
<syntaxhighlight lang="python">

# When training the discriminator
netD.zero_grad()
output = netD(real).to(opt.device)
#D_real_map = output.detach()
errD_real = -output.mean()#-a
errD_real.backward(retain_graph=True)
#... Make noise and prev ...
fake = netG(noise.detach(),prev)
output = netD(fake.detach())
errD_fake = output.mean()
errD_fake.backward(retain_graph=True)

# When training the generator
output = netD(fake)
errG = -output.mean()
errG.backward(retain_graph=True)
</syntaxhighlight>

===Reconstruction Loss===
<math>\mathcal{L}_{rec} = \Vert G_n(0,(\bar{x}^{rec}_{n+1}\uparrow^r) - x_n \Vert ^2</math><br>
The reconstruction loss ensures that the original image can be built by the GAN.<br>
Rather than inputting noise to the generators, they input 
<math>\{z_N^{rec}, z_{N-1}^{rec}, ..., z_0^{rec}\} = \{z^*, 0, ..., 0\}</math>
where the initial noise <math>z^*</math> is drawn once and then fixed during the rest of the training.<br>
The standard deviation <math>\sigma_n</math> of the noise <math>z_n</math> is proportional to the root mean squared error (RMSE) between the reconstructed patch and the original patch.
<syntaxhighlight lang="python">
loss = nn.MSELoss()
Z_opt = opt.noise_amp*z_opt+z_prev
rec_loss = alpha*loss(netG(Z_opt.detach(),z_prev),real)
rec_loss.backward(retain_graph=True)
</syntaxhighlight>

==Evaluation==
They evaluate their method using an Amazon Mechanical Turk (AMT) user study and using Single Image Frechet Inception Distance
===Amazon Mechanical Turk Study===
===Frechet Inception Distance===


==Results==
Below are images of their results from their paper and website.


==Applications==
The following are applications they identify.<br>
The basic idea for each of these applications is to start your input at an intermediate GAN rather than the bottom GAN.<br>
While the bottom layer is a purely unconditional GAN, the intermediate generators are more akin to conditional GANs.
===Super-Resolution===
Upscaling
===Paint-to-Image===
Convert a drawing to an image.
===Harmonization===
Harmonize, or blend the style of a cut-and-pasted piece of image.
===Editing===
===Single Image Animation===
Generate a video from a single image.<br>
In SinGAN, they perform a random walk in the noise passed as inputs to the upper levels of GANs.<br>
* [https://www.youtube.com/watch?v=xk8bWLZk4DU&feature=youtu.be Youtube video]

==Repo==
The official repo for SinGAN can be found on their [https://github.com/tamarott/SinGAN Github Repo]<br>
==Citation==
Here is the bibtex:<br>
<pre>
@InProceedings{Shaham_2019_ICCV,
author = {Shaham, Tamar Rott and Dekel, Tali and Michaeli, Tomer},
title = {SinGAN: Learning a Generative Model From a Single Natural Image},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}
</pre>

[[Category: Papers]]