View source for SinGAN: Learning a Generative Model from a Single Natural Image

SinGAN<br>
[https://arxiv.org/abs/1905.01164 Paper]<br>
[http://webee.technion.ac.il/people/tomermic/SinGAN/SinGAN.htm Author's Website]<br>
[https://tomer.net.technion.ac.il/files/2019/09/SingleImageGan_SM.pdf Supplementary Material]
[http://openaccess.thecvf.com/content_ICCV_2019/supplemental/Shaham_SinGAN_Learning_a_ICCV_2019_supplemental.pdf Mirror(CVF Host)]<br>
[http://openaccess.thecvf.com/content_ICCV_2019/html/Shaham_SinGAN_Learning_a_Generative_Model_From_a_Single_Natural_Image_ICCV_2019_paper.html ICCV]<br>
[https://github.com/tamarott/SinGAN Github Official PyTorch Implementation]<br>
SinGAN: Learning a Generative Model from a Single Natural Image<br>
Authors: 
[https://stamarot.webgr.technion.ac.il/ Tamar Rott Shaham] ([https://www.technion.ac.il/ Technion]), 
[http://people.csail.mit.edu/talidekel/ Tali Dekel] (Google Research), 
[http://webee.technion.ac.il/people/tomermic/ Tomer Michaeli] (Technion)<br>

[[File: Singan_teaser.PNG | 800x400px]]


==Basic Idea==
<!--
This is wrong there are no patches in the code.
Bootstrap patches of the original image and build GANs which can add fine details to blurry patches at different path sizes.
-->
Train GANs to fill in details at different scales of the image
* Start by building a GAN to generate low-resolution versions of the original image
* Then upscale the image and build a GAN to add details to your upscaled image
* Fix the parameters of the previous GAN. Upscale the outputs and repeat.

==Architecture==
They build <math>N</math> GANs, N usually 7-8<br>
Each GAN <math>G_n</math> adds details to the image produced by GAN <math>G_{n+1}</math> below it.<br>
The final GAN <math>G_0</math> adds only fine details.
===Generator===
The use N generators which they call a hierarchy of patch-GANs.<br>
Each generator consists of 5 convolutional blocks:<br>
Conv(<math>3 \times 3</math>)-BatchNorm-LeakyReLU.<br>
They use 32 kernels per block at the coarsest scale and increase <math>2 \times</math> every 4 scales.<br>

====[https://arxiv.org/pdf/1502.03167.pdf Batch Normalization]====
; Definitions:
* Internal Covariate Shift - the change in distribution of network activations as network parameters change.
* Whitening

===Discriminator===
The architecture is the same as the generator.<br>
The patch size is <math>11 \times 11</math>

==Training and Loss Function==
<math>\min_{G_n} \max_{D_n} \mathcal{L}_{adv}(G_n, D_n) + \alpha \mathcal{L}_{rec}(G_n)</math><br>
They use a combination of the standard GAN adversarial loss and a reconstruction loss.

===Adversarial Loss===
They use the [https://arxiv.org/abs/1704.00028 WGAN-GP loss].<br>
This drops the log from the traditional cross-entropy loss.<br>
<math>\min_{G_n}\max_{D_n}L_{adv}(G_n, D_n)+\alpha L_{rec}(G_n)</math><br>
<!--
The final loss is the average over all the patches.<br>
-->
<syntaxhighlight lang="python">

# When training the discriminator
netD.zero_grad()
output = netD(real).to(opt.device)
#D_real_map = output.detach()
errD_real = -output.mean()#-a
errD_real.backward(retain_graph=True)
#... Make noise and prev ...
fake = netG(noise.detach(),prev)
output = netD(fake.detach())
errD_fake = output.mean()
errD_fake.backward(retain_graph=True)

# When training the generator
output = netD(fake)
errG = -output.mean()
errG.backward(retain_graph=True)
</syntaxhighlight>

===Reconstruction Loss===
<math>\mathcal{L}_{rec} = \Vert G_n(0,(\bar{x}^{rec}_{n+1}\uparrow^r) - x_n \Vert ^2</math><br>
The reconstruction loss ensures that the original image can be built by the GAN.<br>
Rather than inputting noise to the generators, they input 
<math>\{z_N^{rec}, z_{N-1}^{rec}, ..., z_0^{rec}\} = \{z^*, 0, ..., 0\}</math>
where the initial noise <math>z^*</math> is drawn once and then fixed during the rest of the training.<br>
The standard deviation <math>\sigma_n</math> of the noise <math>z_n</math> is proportional to the root mean squared error (RMSE) between the reconstructed patch and the original patch.
<syntaxhighlight lang="python">
loss = nn.MSELoss()
Z_opt = opt.noise_amp*z_opt+z_prev
rec_loss = alpha*loss(netG(Z_opt.detach(),z_prev),real)
rec_loss.backward(retain_graph=True)
</syntaxhighlight>

==Evaluation==
They evaluate their method using an Amazon Mechanical Turk (AMT) user study and using Single Image Frechet Inception Distance
===Amazon Mechanical Turk Study===
===Frechet Inception Distance===


==Results==
Below are images of their results from their paper and website.


==Applications==
The following are applications they identify.<br>
The basic idea for each of these applications is to start your input at an intermediate GAN rather than the bottom GAN.<br>
While the bottom layer is a purely unconditional GAN, the intermediate generators are more akin to conditional GANs.
===Super-Resolution===
Upscaling
===Paint-to-Image===
Convert a drawing to an image.
===Harmonization===
Harmonize, or blend the style of a cut-and-pasted piece of image.
===Editing===
===Single Image Animation===
Generate a video from a single image.<br>
In SinGAN, they perform a random walk in the noise passed as inputs to the upper levels of GANs.<br>
* [https://www.youtube.com/watch?v=xk8bWLZk4DU&feature=youtu.be Youtube video]

==Repo==
The official repo for SinGAN can be found on their [https://github.com/tamarott/SinGAN Github Repo]<br>
==Citation==
Here is the bibtex:<br>
<pre>
@InProceedings{Shaham_2019_ICCV,
author = {Shaham, Tamar Rott and Dekel, Tali and Michaeli, Tomer},
title = {SinGAN: Learning a Generative Model From a Single Natural Image},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}
</pre>