Image by Free-Photos from Pixabay

THOUGHTS AND THEORY

Google’s RFA: Approximating Softmax Attention Mechanism in Transformers

What is Attention Mechanism & Why is RFA better than Softmax?

Google has recently released a new approach — Random Feature Attention — to replace softmax attention mechanisms in transformers for achieving similar or better performance with…