Recurrent Neural Networks for Language Understanding
Recurrent Neural Networks (RNNs) have been credited with achieving state of the art performance in machine translation, sentiment analysis, speech recognition and many other machine learning tasks. Their strengths lie in their ability to process sequential data and outputs and inputs of various lengths.
These miraculous networks aren’t anything new. In fact, they were developed in the 1980s — but being more computationally costly than non-recurrent neural networks, it wasn’t until (relatively) recent improvements in computational resources that RNNs took off in popularity.
This article will introduce Recurrent Neural Networks in the context of NLP.
Learning from Sequential Data
Feed Forward Neural Networks make the assumption that all inputs are independent of each other. While this is a fair assumption to make for many classification tasks, it fails to capture relationships found in sequential data.
In sequential data, the order matters. Take text for example. We develop our understanding of text with every word. We must process text in sequence and each word in conjunction with what was learned before to form a proper understanding.
Recurrent Neural Networks mitigate this problem by using previously learned information to predict the next output. In the context of language, RNNs store information learned from previous words in “memory” and uses this knowledge to understand…