New NN Checklist

Checklist that should be considered when building new Neural Network:

  • Define/validate your data
  • Preprocess the data
  • Split data on train/validation/test sets
  • Make small network
  • Validate gradient if it was manually coded
  • Try to use pretrained weights
  • Measure accuracy/ loss
  • Calculate overall quantity of trainable parameters and memory consumption
  • Use SGD as first approach without momentum. After give a try to Nesterov momentum and Adam optimizer.
  • Try to overfit with small amount of data
  • Validate data precise
  • Add more metrics
  • There should be clear metric that allow you to compare networks with different params.
  • Add regularization to loss. Compare it with cross-entropy loss
  • Try to find optimal parameters with random search or grid search
  • If optimal params exist on the edge of available range it is important to double check larger range
  • Add some bells and whistles. Your network should work fine without them also