Machines will change the world, they say. They will empower humans to achieve things beyond our imagination, solving incredibly complex conundra in data interpretation and projecting us into a future of predictive diseases and infallible trading strategies. They might not be great at everything (yet), but it surely is just a matter of time.
It turns out that much-praised Machine Learning tends to be not as good as human learning in things where humans are spectacularly good at already. The ultimate purpose of deep learning, defined as the tentative to solve a non-linear optimisation problem through the training of stacked layers of neurons (or perceptrons) under human guidance, has been exactly that of trying to bridge the performance gap between humans and machines. Be as good as us in recognising a person, labelling a picture (ConvNets), sensing emotions, guessing sentiment, decoding language, or producing art (RNNs).
But that is just one part of the story I’m sure. I read somewhere, probably in the brochure of some expensive data management consulting, that although machines are not naturally much better than humans in so-called perception problems, they are indeed phenomenal when it comes to making sense of complexly structured information: optimisation problems that require iterations among millions of pieces of data, from innumerable sources. This must be true of incredibly powerful machines.
However, not even the latter is such a clear case, and it doesn’t take much effort to hit the ceiling of what the computational power of the most modern machine can do. In the article below I argue that knowing the limits of machine learning is more interesting and instructive than trying to worship it as the panacea of our century. I hope to be worth of your attention.
Are machines actually good at what they are doing?
Some problems can’t really be decided logically. Every age has its paradoxes: problems that puzzle logicians trying to patch axioms, procedures, interpretations, and solutions. There is today a long list of computable yet unsolvable problems where a deterministic machine, equipped by a functioning algorithm and infinite time and memory, couldn’t give you a consistent answer without getting trapped into an infinite loop. Turing’s Halting Problem being one of the most famous in computational theory and mathematical philosophy. This is all material for a series of posts that hopefully I will find the ability to write one day, but for now let’s overlook paradoxes assuming there might be something wrong in their definition or tentative solution, and move on.
Unfortunately, even within the realm of clearly statable, understandable, computable problems, you don’t need to get too far to hit the limits of what even a super-powerful machine could actually do.
Imagine you are a travelling salesman, looking at a map with names of cities and distances and trying to figure out the shortest possible route that visits each city exactly once and returns home. That would be an easy optimisation problem to be solved by a modern computer. Not really. In Mathematics such problems are called NP-hard (Non-deterministic Polynomial-time hard – this is also material for another post) problems of combinatorial optimisation, meaning that they are very, very costly to solve. In 2020, a Japanese supercomputer celebrated the solution of a Travelling Salesman Problem (or TSP) involving 22 cities, improving from the previous 16 cities record. 22 cities, that’s all. It is so difficult because solving the TSP with brute force means solving a problem of superpolinomial (or exponential) complexity. Optimising the salesman’s path around 22 cities would require 22! (factorial – 22x21x20x[…]x1) steps. If one step would require 0.001 seconds to be computed by a machine, finding the solution would take only a bit less than 36 billion years. Our universe has existed for 14 billion years so far, and it is expected to stay around for another 5 by some physicists (although the judge isn’t yet out on this one). There simply isn’t enough time to help the salesman.

To complicate matters further, there are other problems that are very difficult to solve from scratch, but that can have a suitable solution easily checked. The most famous of those is the Dormitory Problem. Suppose you are tasked with organising accommodation for an incoming class of 400 college students. However, space is limited and you will be able to host on campus only 100 of them. In addition, the Dean has given you a list of pairs of incompatible students you must avoid in your configuration. Although it seems easy at first, the task of generating a list from scratch is truly impractical (non-P), whereas that of checking whether a list is compliant with guidance is simply trivial (NP).
Effectively, the total number of ways of choosing 100 students from the 400 applicants is greater than the number of atoms in the known universe.
Today’s mathematicians and logicians believe we have plenty of those "easy to check / difficult to solve" (P ≠ NP) problems, although so far nobody has proven that such inequality holds. If you believe in your arguments, you can apply to receive the 1 million USD grant here.

Surprisingly, it turns out humans are extremely good at solving this type of super-difficult optimisation problems. It has been observed that humans are able to produce a quick solution for the TSP with performance ranging within 1% from the computed optimum for graphs of 10–20 nodes, and within 11% from the (estimated) solution for 120 nodes. Pigeons are pretty good at this too apparently; those improbable super-optimizers seem empowered, rather than with incredibly powerful computational minds, by a mix of spatial heuristics and experiential reinforcements that make them very effective without wasting too much time on the nitty-gritty.
The similarities with the architecture and mechanics of neural networks are obvious. Although it might not be proven that the solution to the optimisation problem offered by the machine coincides with the global optimum (if that multi-dimensional point exists), we have developed a set of techniques to induce enough randomness in the process to shake things up and reduce the risk of being trapped in local optimality (there is a plethora of search methods out there for you to try). We have, de facto, trained a machine to think, or rather act, imperfectly like humans – i.e. through an educated process of trial and error which has been built on tradition (read supervised learning), present enjoyment or suffering (read reinforcement), and continuous innovation (read stochastic techniques).
So the engineers created algos in their own image
It has been empirically found that AI is highly dependent on the knowledge of the human professionals that spend their lives in the very work practices subject to automation. But it is exactly this use of human knowledge that makes it difficult for a machine to reach performance levels that are beyond those of such humans.
Total Error to assess the performance of a model can be separated into three more elementary components:
- Bias: model error or oversimplification – the machine cannot extract from the training set the relevant characteristics that allow to predict labels correctly during both training and testing
- Variance: generalisation error or overfitting – the machine seems to do very well in predicting labels in training, but performs poorly when generalising for the data set
- Irreducible (Bayes) Error: the unavoidable error – unless a process is truly deterministic, there will always be noise in the data, or an irreducible error that would occur even in presence of perfect information (have you ever read about quantum physics)
Continuing along our analogy for classification problems, human engineers are (i) the maestro of reducing Bias (machine architects and providers of categorised training data), the judge of assessing Variance (evaluators of the ultimate output quality), and the gold standard for Irreducible Error (the best proxy for irreducible error is, surprise, trained human error).
You should still get worried, or excited
Don’t get me wrong, the rise of the machines is far from being irrelevant. If you believe (deep inside yourself) that any inexperienced apprentice could be easily trained to do your job sufficiently well, a machine will get soon pretty close in performing the same task. Close enough to let your employers ask themselves whether it does make sense to substitute some of us for an algorithm. And this is because machines, differently from humans, have several traits a perspective employer would appreciate in an interview: they are incredibly fast and don’t need rest, have reasonably predictable incentives, don’t complain, their supply for specific tasks doesn’t take a whole school cycle to be expanded or reduced, and they don’t want to get paid (yet – the crypto crowd would disagree with me).
Nevertheless, I am convinced that understanding the limitations of how machines perceive and deal with reality could tell us more than what we think about ourselves as humans, and about the forces that in turn have shaped us during the past several billion years. Ultimately, it is us who created them in our own image.