A Comprehensive Guide of Neural Networks Activation Functions: How, When, Why?

Picture generated from Stability Diffusion using the title as prompt

In artificial neural networks (ANN), the units are designed to be a loose copy of biological neurons in the brain. The first artificial neuron was designed by McCulloch and Pitts in 1943 [1] and consisted of a linear threshold unit. This was to mimic that artificial neurons do not just output the raw input they receive, but instead, output the result of the…