View source for Parallel Neural Network Training

Note: I do not do this. This is just a collection of resources.

==Types of Parallelism==
See [https://arxiv.org/pdf/2101.03961.pdf Switch Transformer Paper]

===Data Parallelism===
Here, the entire model is copied on each GPU. Each GPU works on a different subset of the batch.  
Gradients are accumulated together to update the model.

===Model Parallelism===
Here, different parts of the model are distributed across different GPUs.