PyTorch: Difference between revisions
| (8 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
==Installation== | ==Installation== | ||
See [https://pytorch.org/get-started/locally/ PyTorch Getting Started] | See [https://pytorch.org/get-started/locally/ PyTorch Getting Started] and [https://pytorch.org/get-started/previous-versions/ PyTorch Previous Versions] | ||
I recommend using the conda installation method since it is paired with the correct version of cuda. | |||
==Getting Started== | ==Getting Started== | ||
* [https://pytorch.org/tutorials/ PyTorch Tutorials] | * [https://pytorch.org/tutorials/ PyTorch Tutorials] | ||
{{hidden | Example | | |||
<syntaxhighlight lang="python"> | <syntaxhighlight lang="python"> | ||
import torch | import torch | ||
import torch.nn as nn | import torch.nn as nn | ||
model = nn.Sequential(nn.Linear(5, 5),nn.ReLU(),nn.Linear(5, 1)) | |||
criterion = nn.MSELoss() | |||
optimizer = torch.optim.Adam(model.parameters(), lr=0.001) | |||
# Training | # Training | ||
for epoch in range(epochs): | for epoch in range(epochs): | ||
for i, data in enumerate(trainloader): | |||
for i, data in enumerate(trainloader | # get the inputs; e.g. data is a list of [inputs, labels] | ||
# get the inputs; data is a list of [inputs, labels] | |||
inputs, labels = data | inputs, labels = data | ||
| Line 26: | Line 27: | ||
optimizer.zero_grad() | optimizer.zero_grad() | ||
# forward | # forward | ||
outputs = | outputs = model(inputs) | ||
loss = criterion(outputs, labels) | loss = criterion(outputs, labels) | ||
# backward | |||
loss.backward() | loss.backward() | ||
optimizer.step() | optimizer.step() | ||
</syntaxhighlight> | </syntaxhighlight> | ||
}} | |||
==Importing Data== | ==Importing Data== | ||
| Line 38: | Line 42: | ||
==Usage== | ==Usage== | ||
Note that there are several useful functions under <code>torch.nn.functional</code> which is typically imported as <code>F</code>. | |||
Most neural network layers are actually implemented in functional. | |||
===torch.meshgrid=== | ===torch.meshgrid=== | ||
Note that this is transposed compared to <code>np.meshgrid</code>. | Note that this is transposed compared to <code>np.meshgrid</code>. | ||
===torch. | ===torch.multinomial=== | ||
[https://pytorch.org/docs/stable/ | [https://pytorch.org/docs/stable/generated/torch.multinomial.html torch.multinomial]<br> | ||
====F.grid_sample | If you need to sample with a lot of categories and with replacement, it may be faster to use `torch.cumsum` to build a CDF and `torch.searchsorted`. | ||
{{hidden | torch.searchsorted example | | |||
<syntaxhighlight lang="python"> | |||
# Create your weights variable. | |||
weights_cdf = torch.cumsum(weights, dim=0) | |||
weights_cdf_max = weights_cdf[0] | |||
sample = torch.searchsorted(weights_cdf, | |||
weights_cdf_max * torch.rand(num_samples)) | |||
</syntaxhighlight> | |||
}} | |||
===F.grid_sample=== | |||
[https://pytorch.org/docs/stable/nn.functional.html#grid-sample Doc]<br> | [https://pytorch.org/docs/stable/nn.functional.html#grid-sample Doc]<br> | ||
This function allows you to perform interpolation on your input tensor.<br> | This function allows you to perform interpolation on your input tensor.<br> | ||
| Line 53: | Line 71: | ||
* In your class include all other modules you need during init. | * In your class include all other modules you need during init. | ||
** If you have a list of modules, make sure to wrap them in <code>nn.ModuleList</code> or <code>nn.Sequential</code> so they are properly recognized. | ** If you have a list of modules, make sure to wrap them in <code>nn.ModuleList</code> or <code>nn.Sequential</code> so they are properly recognized. | ||
* Wrap any parameters for you model in <code>nn.Parameter(weight, requires_grad=True)</code>. | |||
* Write a forward pass for your model. | * Write a forward pass for your model. | ||
| Line 58: | Line 77: | ||
See [https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html Multi-GPU Examples]. | See [https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html Multi-GPU Examples]. | ||
==nn.DataParallel== | ===nn.DataParallel=== | ||
The basic idea is to wrap blocks in [https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel <code>nn.DataParallel</code>]. | The basic idea is to wrap blocks in [https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel <code>nn.DataParallel</code>]. | ||
| Line 103: | Line 122: | ||
===Float16=== | ===Float16=== | ||
Float16 uses half the memory of float32. | Float16 uses half the memory of float32. | ||
New Nvidia GPUs also have dedicated hardware called tensor cores to speed up float16 matrix multiplication. | New Nvidia GPUs also have dedicated hardware instructions called tensor cores to speed up float16 matrix multiplication. | ||
Typically it's best to train using float32 though for stability purposes. | Typically it's best to train using float32 though for stability purposes. | ||
You can do truncate trained models and inference using float16. | You can do truncate trained models and inference using float16. | ||
| Line 155: | Line 174: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
==PyTorch3D== | ==Libraries== | ||
A list of useful libraries | |||
===torchvision=== | |||
https://pytorch.org/vision/stable/index.html | |||
Official tools for image manipulation such as blur, bounding boxes. | |||
===torchmetrics=== | |||
https://torchmetrics.readthedocs.io/en/stable/ | |||
Various metrics such as PSNR, SSIM, LPIPS | |||
===PyTorch3D=== | |||
{{main | PyTorch3D}} | {{main | PyTorch3D}} | ||
[https://github.com/facebookresearch/pytorch3d PyTorch3D] | [https://github.com/facebookresearch/pytorch3d PyTorch3D] | ||
Facebook library with differentiable renderers for meshes and point clouds. | |||