Self-Attention In Computer Vision

Ever since the introduction of Transformer networks, the attention mechanism in deep learning has enjoyed great popularity in the machine translation as well as NLP communities. However, in computer vision, convolutional neural networks (CNNs) are still the norm and self-attention just began to slowly creep into the main body of research, either complementing existing CNN architectures or completely replacing them. In this post I…