Magic boxes & machine learning— why we need to stop using the black box metaphor
The author Ocean Voung recently posted about metaphors and described the work that they do beyond description. He talks about how a well-constructed metaphor leads the audience to further discoveries in the story or successfully amplifies the meaning of what is there, beyond just adding texture to an existing description.
We use metaphors all the time when we’re tealking about technology and data. The images we choose reveal how we think about and understand it, often doing the work Vuong is talking about. For example, people who like the ‘data as oil’ metaphor tend to be the kinds of people who like extracting and profiting from oil.
Nick Seaver’s ethnography of music recommendation engineers ‘Parks and Recommendation: Spatial Imaginaries in Algorithmic Systems’ (😏) shows how they discuss their work through spatial metaphors. They describe their roles as being like park rangers (who enable people to follow their own musical paths), or gardeners (taming the wilderness of music metadata to create orderly beds of thematically arranged blooms). The two metaphors represent very different ways of discovering and listening to music and the team actively use the images to describe and engage with the various structures they develop and embed in the software, not just as passive descriptors. The metaphors are part of the engineering process.
One of the most pervasive and widely used metaphors we use about modern technology is the ‘black box’, to the extent it has become a cliche. It refers to machine learning models that employ unsupervised learning techniques meaning their outputs can’t be reverse-engineered — you can’t ‘open’ them up to see what processes have been applied to their inputs. For example, in the case of GPT-3 — raw data does in, and natural language comes out. It appears magical.
Magic boxes are not new ideas- they are littered through folklore in one form or another. Taking a quick look down the helpful ‘containers’ subsection of this list of mythological objects (thanks Wikipedia), you can see that most magic receptacles contain a combination of never-ending food, evil spirits, wronged gods, secret weapons, and special knowledge. In the stories, the boxes provide a narrative function, a mechanism for moving the story along by giving the hero the secret weapon or unleashing a great threat. So when we use the black box metaphor for technology it does more than just describe an opaque technical process, it also brings with it that lineage of mythological connotations.
Advocates for unsupervised learning present it in a way that invokes Yorroba tales of the ‘Basket of Existence’, a raffia basket containing the divine ingredients that were to be used by Obatala to create the universe. Those that distrust the technology regard it more as a Pandora’s box waiting to unleash the singularity. Ad tech‘s ability to serve ads with such uncanny accuracy it makes people feel they’re being covertly listened to is reminiscent of the ‘Purple Gold Red Gourd’ from Chinese mythology- a powerful magic gourd that sucks anyone who speaks before it inside and melts them down into a bloody stew (🤢).
For engineers, the illusion the black box metaphor provides is that with enough training data, regardless of where it comes from, one can statistically override any complex, messy, social contradictions represented in the training data and generate objective outputs. In this way engineers seem to treat black boxes more like a TARDIS or the box Schrödinger’s cat is stuck in- a space that contains a paradox, maintains a bubble of metaphysical incongruence, and enables seemingly impossible things like time travel or zombie cats.
Unfortunately, the technology has outgrown its first flourish of magical realism. Applications using unsupervised learning trained on vast amounts of closed data are now regularly enabling as much social harm as they are social value. Outputs that are clearly far from objective or neutral and researchers are starting to point out the smoke and mirrors upholding the illusion.
The now infamous paper written by Timnit Gebru which lead to her departure from Google is said to demonstrate the dangers of massive natural language models like GPT-3. According to those who have read the paper, it makes the point that training models on vast amounts of user-generated text content, much of which will be scrapped of Reddit and other sources, will inevitably lead to some racist, sexist, and otherwise abusive language training the model. Beyond the fact that an AI model taught to view racist language as normal is clearly not ok, this method cannot account for the subtleties of shifting language which is particularly relevant around social change movements such as MeToo and Black Lives Matter. Similarly, such models won’t be able to grasp the language and norms of different countries and cultures, resulting in a homogenisation of online linguistics that centre around western, privileged practices. To summarise the summaries of the paper:
- Every data set has a setting that has to be acknowledged.
- Language matters and the words we use have power, particularly when they are used to build AI technologies.
Whilst of course Gebru and her co-authors are talking about the language in training data, there is also value in thinking about the language we use to describe the technology itself. It may seem trivial but if a well-constructed metaphor can lead to further discoveries or amplify the meaning of what is there, then surely it’s worth looking for images that are indicative of the nature of the tech as it actually is, or how we want it to be.
Loath as I am to invoke the cliche of ‘breaking out of the box’, there’s got to be a better metaphor.
