In this article I will show that deep learning is incapable of understanding logic and structure, and point to a potential solution inspired by neuroscience. This is important since most worthwhile problems in the world need to be solved logically, but modern deep learning largely failed in that department.
What does it mean to be logical?
Logics, in statistical terms, is analogous to extreme confidence in a prediction given a set of observations. In a purely logical world, 1 + 1 = 2 is always true, whereas in a more random, non-logical system, 1 + 1 = 2 might be true only 90% of the time.
Some logics arise from our understanding of the structure of the world, such as laws of physics. For example, if you drop a ball and then you expect it to fall to the ground. In an unstructured world, however, anything is possible, rendering it difficult to predict the future. For example, stock market is an unstructured world. My father has been watching over Novax Pharma since May, but he couldn’t have predicted that the CEO would sell out his stock and make its price plummet.
Statisticians might claim that we live in a random and unstructured world, but there are also many aspects of the universe that are predictable and logical, such as physics, mathematics, and science. Being logical allows us to plan far into the future, and form concrete paths that help us reach our goals. Almost all difficult problems that are worth solving need to use reasoning and logic.
Pure deep learning cannot learn logic
Can Deep Learning achieve the holy grail of logic? DeepMind asked this question in their 2019 paper where they implemented a transformer model to solve math problems [1]. Their result was impressive; the model reached higher than 90% accuracy in simple addition, subtraction, division, and multiplication. But the performance dropped to 50% when the operations were mixed, which suggested that the model was just guessing the solution rather than solving the problem step-by-step.
There are other examples in deep learning where the models or the agents are so adept at their tasks that they generate illusions of logic and reasoning.
OpenAI’s GPT-2 language model can generate human-like essays, but closer inspection shows that its outputs are not logically constructed and it simply mimics what it was trained to regurgitate in the training data. For example, the model sometimes writes about fires happening under water.
The video game agents developed by DeepMind and OpenAI to play StarCraft 2 and DotA 2, respectively, defeated top professional gamers in their respective games, but mostly using sheer mechanical skills and well executed attacks, not strategies. For example in DotA 2, there is an action that a player can take that instantly kills another player if the health of the target is below a threshold, but little to no damage otherwise. Using it above the threshold is undeniably a mistake, but OpenAI’s DotA 2 bot does it all the time. AlphaStar, DeepMind’s AI for StarCraft 2, consistently tilts and loses games when human opponents use obscure strategies that cannot be overcome by mechanical skill alone.
Some argue that these agents might overcome the above flaws if they are trained longer. This may be true, but these issues typically do not manifest in even average human players. It is clear that the agents are missing an ingredient that is yet to make them as intelligent as human.
Neuroscience gives us the answer
Back in February I stumbled upon the interview by Lex Fridman with theoretical neuroscientist Jeffrey Hawkins. In the interview, Hawkins described that neuroscientists are hypothesizing that the neural mechanism for spatial navigation in human could also be responsible for human’s ability to navigate abstract ideas. I mean, why not? Solving problems logically uses the same principle as spatial navigation in that both require planning a route from the origin to the destination.
In 2018, DeepMind happened to implement the world’s first agent that used the neural substrate for spatial navigation (called Grid Cells) [2]. The agent was tasked to navigate maze-like environments, and the involvement of grid cells taught the agent to take short-cuts all the time and create new short-cuts after the original short-cuts were blocked. This is an incredible feat, something the original discoverer of grid cells reviewed as "notoriously difficult in robotics". Most importantly, however, their experiments found that the neural network used in the agent developed grid cell-like properties by just having the network estimate where the agent was located and facing in the maze every time it moved. To put it simply, their finding suggested that self-awareness (in this case spatial self-awareness) is a key ingredient for solving any kind of navigational problem. This shouldn’t come as a surprise for most of us, since assessing where we are relative to goals is crucial for achieving them.
Here is the bombshell idea of this article. We have been training deep learning models by minimizing error of prediction (ex, classifying whether an image is that of a cat, or a dog, etc…), but would it be possible for the models to understand the structure of this world and navigate abstract ideas if we minimize its error on self-awareness?
Implementing self-aware AI
Assuming that self-awareness is one of the necessary ingredients for AI to solve problems logically, how would we even implement it? There are some takeaways from the grid cell paper:
- The error on "self-awareness" was the difference between the ground-truth and the agent’s own predicted location and orientation.
- The ground-truth location and orientation (the self-awareness of the agent) were represented by neural activation signatures, each unit of which fires when the agent is at a unique location and orientation.
- Grid cells are activated at regular intervals relative to where the agent is located and how it is orientated
- Grid cell activations emerge just prior to the final linear layer as the model trains to minimize the loss on self-awareness.
To summarize, "self-awareness" will simply fall out of training as long as we define the environment which it exists in, and minimize the error on the prediction of its coordinate and orientation. Unfortunately, experiments involving grid cell has only been done on spatial navigation, so it is unclear whether it could be adapted to a non-spatial system.
But I do have an experiment in mind. A hot-topic of research in NLP is to understand how to teach a model to capture cause-and-effect relationship. The earlier example I mentioned regarding GPT-2 writing about fires happening under water is an example of the model confusing cause-and-effect with correlation. Just because there are sentences that says water frequently extinguishes fire doesn’t mean fire influences water. Would grid cells that learn to navigate the vector space of word embeddings better capture this relationship?
Conclusion
While the above experiment is speculative and this article could be a complete false alarm, it is also undeniable that deep learning has hit a wall, and a worthwhile way forward for the research community is to explore new ideas (gee, thanks captain obvious!). During my computer science thesis project I explored the potential of knowledge graph in imparting neural network the ability to reason. It is a cool concept, but this project failed due to the intrinsic limitation of graphs.
As of writing this article, I am taking a break from deep learning to focus on my last year of education, so I am passing on the torch to you. For those of you who are tired of optimizing existing architectures or are attracted to this idea, I heavily encourage you to look into DeepMind’s paper on grid cell and adapt it to a non-spatial application. Who knows, you might discover a new architecture that performs better than the existing ones. But if you are, just remember that you heard it from here first.
[1] D. Saxton, E. Grefenstette, F. Hill, and P, Kohli, Analysing mathematical reasoning abilities of neural models (2019), ICLR 2019
[2] A. Banino, et al. Vector-based navigation using grid-like Representations in Artificial Agents (2018), Nature 26, 1.