The Subtle Art of Fixing and Modifying Learning Rate

An introduction to learning rate hyper-parameter and principles and procedures to find a good starting value and adapting it over the course of training.

Picture from internet

Learning rate is one of the most critical hyper-parameters and has the potential to decide the fate of your deep…