5,323
edits
PyTorch: Difference between revisions
→Memory Usage
| Line 67: | Line 67: | ||
The main difference is this uses multiple processes instead of multithreading to work around the Python Interpreter. | The main difference is this uses multiple processes instead of multithreading to work around the Python Interpreter. | ||
==Memory | ==Memory== | ||
Reducing memory usage | ===Reducing memory usage=== | ||
* Save loss using [https://pytorch.org/docs/stable/tensors.html#torch.Tensor.item <code>.item()</code>] which returns a standard Python number | * Save loss using [https://pytorch.org/docs/stable/tensors.html#torch.Tensor.item <code>.item()</code>] which returns a standard Python number | ||
* For non-scalar items, use <code>my_var.detach().cpu().numpy()</code> | * For non-scalar items, use <code>my_var.detach().cpu().numpy()</code> | ||
| Line 75: | Line 75: | ||
* <code>cpu()</code> copies the tensor to the CPU | * <code>cpu()</code> copies the tensor to the CPU | ||
* <code>numpy()</code> returns a numpy view of the tensor | * <code>numpy()</code> returns a numpy view of the tensor | ||
When possible, use functions which return new views of existing tensors rather than making duplicates of tensors: | |||
* [https://pytorch.org/docs/stable/tensors.html#torch.Tensor.permute <code>permute</code>] | |||
* [https://pytorch.org/docs/stable/tensors.html#torch.Tensor.expand <code>expand</code> (as opposed to <code>repeat</code>)] | |||
* [https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view <code>view</code>] | |||
Note that <code>permute</code> does not change the underlying data. | |||
This can result in a minor performance hit if you repeatedly use a contiguous tensor with a channels last tensor. | |||
To address this, call [https://pytorch.org/docs/stable/tensors.html#torch.Tensor.contiguous <code>contiguous</code>] on the tensor with the new memory format. | |||
==TensorBoard== | ==TensorBoard== | ||