Unpopular Opinion: We'll Abandon Machine Learning as Main AI Paradigm

ARTIFICIAL INTELLIGENCE

Machine learning and deep learning will slowly lose their status until they get relegated to what they truly are; fancy statistical techniques.

AI has been dominated by connectionist AI – neural network-based AI – for at least two decades. From recognizing handwritten digits to mastering language, breakthroughs have been occurring one after the other. AI has been advancing so fast, the world hasn’t been able to keep pace. Despite the popularity of the field, one of the leading pioneers of the neural network revolution, Geoffrey Hinton – Godfather of AI -, thinks we should rethink everything from scratch: "My view is throw it all away and start again."

Machine learning (ML) and deep learning (DL), the current leading AI paradigms, have done a fantastic job so far. A good example is the recent boom of transformer-based language models – like GPT-3. But there are obstacles ahead that seem to be out of reach for today’s approaches. There are nascent frameworks that will take over AI in the coming years. Here are 2 reasons why we’ll abandon connectionist AI as the leading force driving the field forward.

1. We won’t reach AI’s goal with current paradigms

AI can be looked at from two very different perspectives.

First, the practical, applicable aspect of AI. We’re solving problems with deep learning methods we thought impossible just a few years ago. Object/speech detection and recognition at human levels, creative systems, conversational bots, or language masters. From this point of view AI looks fine.

But there’s another perspective: Looking at the direction we’re heading. And it doesn’t look that good. There’s an ongoing debate in the field. No one knows which is the right path to reach artificial general intelligence (AGI). Deep learning-based solutions are okay, but the ultimate goal – which could hypothetically solve everything – is, and always has been, AGI. Experts agree about the state-of-the-art as much as they disagree on what the next step should be.

Some say we’re heading in the right direction with larger, more powerful language models – which is fair given the success of GPT-3 and its successors. Others say we have to make some twists and turns here and there – think fully self-supervised models, reinforcement learning, hybrid models. And others argue we have to add new things that don’t exist yet – such as system 2 reasoning, causality, or intuitive physics.

We’re the only instance of high intelligence we know of. It’s fair to assume AI will resemble some of the features that make it so, at least to some degree. That’s why increasingly more researchers defend the idea of embodied AI: Intelligence can’t be acquired without a body that interacts with the world. Alva Noë argues in his book Action in Perception that "perception is not a process in the brain, but a kind of skillful activity of the body as a whole." Our intelligence arises from the way we grow, live, and interact in the world.

The sub-field of developmental robotics aims at building AI machines in the form of physical robots that could grow as a human child does. But this isn’t a new idea; Turing himself argued that "instead of trying to produce a programme to simulate the adult mind, [we could] rather try to produce one which simulates the child’s." We could then divide the problem into two better-defined parts: Building a child’s brain – easily programmable in Turing’s hypothesis – and then educate it.

This solution is completely out of reach for today’s ML/DL. Virtual AIs can solve some problems but not all. Because ML- and DL-based AIs don’t live in the real world, they can’t interact with it the way we do it. And therefore they won’t ever become as intelligent as we are.

2. We’re approaching AGI by taking shots in the dark

But let’s give ML and DL the benefit of the doubt and assume we could build AGI by continuing this path.

We’ve been building bigger models, trained with more data in bigger computers since interest in DL skyrocketed in 2012. This idea of "bigger is better" has found important successes across the sub-fields of natural language processing and computer vision, among others. As long as we’re able to develop larger models, this approach will probably keep giving us better results. The hope is that sometime in the Future, one of those models gets so intelligent that it reaches the status of AGI – we aren’t even close now.

GPT-3 is a good example of this attitude. It became the larger neural network ever created at the whooping mark of 175 billion parameters – it was 100x bigger than its predecessor, GPT-2. It showed off, performing at the top level in many different language tasks and even tackling problems previously reserved to us, like writing poetry or music, transforming English to code, or pondering about the meaning of life.

GPT-3 was so much more powerful than other models that we soon found ourselves unable to assess its limitations: The authors hadn’t thought about many of the use cases people found. People kept trying to find its weaknesses, once and again crashing into the wall of their own limitations. GPT-3’s power was outside the limits of our measurement tools. Whatever its level of intelligence, what we were measuring was below that.

Another example is Wu Dao 2.0, which was released a month ago – now holding the record of largest neural network ever created, which it will lose in no time. This monster of 1.75 trillion parameters was 10x bigger than GPT-3. We couldn’t adequately measure GPT-3’s level of intelligence – although is generally accepted it isn’t AGI-level – and we keep building yet larger and larger models.

Why are we approaching AGI this way? Where is this leading us?

We’re taking shots in the dark. The financial benefit is a tempting objective to go after, and that’s exactly what most of the companies and institutions behind these models are fighting for. What would happen if we keep building larger models which intelligence we can’t assess?

By making the assumption I described at the beginning of this section, we conclude we’ll eventually build an AGI-level system using current techniques. We’ll be looking for it and we’ll find it. But we won’t know it because the tools that we use to define our reality will be telling us another story. And, because we’re walking forward with the lights off, we won’t even pause for a second and doubt whether it has happened.

How dangerous is this scenario? We’re trying to build the most powerful entity ever. We’re doing it in the dark. We’re doing it mindlessly. We’re doing it because of money and power. If, in the end, ML and DL could create AGI, we better find a way to avoid this scenario. We should shift both our mentality and the paradigms – to others more interpretable and more responsible.

Final thoughts

ML and DL are synonyms of AI today. Every major breakthrough in the field in the last 15–20 years is due to the unreasonable effectiveness of these approaches. They work wonders to solve some problems, but they aren’t the panacea; they won’t solve the ultimate problem of AI.

We either will replace these paradigms with other, better-suited paradigms that are indeed able to keep moving the field forward past the obstacles of connectionist AI. Or we’ll better remove them from being the main force forward unless we want to crash into AGI without knowing it and potentially find ourselves in the deeply feared scenario in which AI overthrows and overrules us.

Let’s de-romanticize machine learning and deep learning. Let’s try to not ask them more than they can give. They are extremely useful paradigms, but History has shown us once and again that even what looks like the final solution often isn’t.