Debugging ML Models

Notes on debugging ML models, primarilly CNNs.

Debugging

• Train on a single example and see if it overfits.
  • If it doesn't overfit, there may be an issue with your code.
  • You can try increasing the capacity (e.g number of filters or number of nodes in FC) 2-4x.

    If the input is 3 channels, then the first conv layer should have more than 3 channels.

  • Check that your loss is implemented correctly and taken against the correct ground truth image.
• Dump all inputs and outputs into TensorBoard. You may have an unexpected input or output somewhere.
• Make sure there is no activation on the final layer.

Underfitting

If it looks like it is underfitting (e.g. if the training output and validation output are both blurry), then you can try the following.

• Train for 4x as long until the training loss and validation loss both flatten.
• Increase or decrease the learning rate one magnitude.
• Make sure the batch size is a multiple of 2. Try increasing it to get more stable gradient updates or decreasing it to get faster iterations.
• Try disabling any tricks you have like dropout.

Overfitting

Overfitting occurs when your training loss is below your validation loss.
Historically this was a big concern for ML models and people relied heavily on regularization to address overfitting.
Recently though, overfitting has become less of a concern with larger ML models.

• Increase the capacity and depth of the model.
• Add more training data if you can.
• Depending on your task, you can also try data augmentation (e.g. random crops, random rotations, random hue).
  • PyTorch has many augmentations in torchvision.transforms.
  • TF/Keras has augmentations as well in tf.image. See Tutorials: Data Augmentation.