Debugging ML Models: Difference between revisions
No edit summary |
|||
Line 8: | Line 8: | ||
** Check that your loss is implemented correctly and taken against the correct ground truth image. | ** Check that your loss is implemented correctly and taken against the correct ground truth image. | ||
* Dump all inputs and outputs into [[TensorBoard]]. You may have an unexpected input or output somewhere. | * Dump all inputs and outputs into [[TensorBoard]]. You may have an unexpected input or output somewhere. | ||
* Try disabling any tricks you have like dropout. | * Try disabling any tricks you have like dropout. | ||
* Make sure there is no activation on the final layer. | * Make sure there is no activation on the final layer. | ||
==Underfitting== | |||
If it looks like it is underfitting (e.g. if the training output and validation output are both blurry), then you can try the following. | |||
* Train for 4x as long until the training loss and validation loss both flatten. | |||
* Increase or decrease the learning rate one magnitude. | |||
* Make sure the batch size is a multiple of 2. Try increasing it to get more stable gradient updates or decreasing it to get faster iterations. |