10 Stochastic Gradient Descent Optimisation Algorithms + Cheatsheet
Stochastic gradient descent optimisation algorithms you should know for deep learning
Published in
12 min readNov 22, 2018
(I maintain a cheat sheet of these optimisers including RAdam in my blog here.)
Changelog:
5 Jan 2022 — Fix typos
4 May 2020 — Fix typo in Nadam formula in Appendix 2
21 Mar 2020 —…
