AI Alignment and Safety
It started with a late night Tweet:
<iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&key=a19fcc184b9711e1b4764040d3dc5c07&schema=twitter&url=https%3A//twitter.com/bellmar/status/1443420485816143872&image=https%3A//i.embed.ly/1/image%3Furl%3Dhttps%253A%252F%252Fabs.twimg.com%252Ferrors%252Flogo46x38.png%26key%3Da19fcc184b9711e1b4764040d3dc5c07" title=""Alarmed by how many AI/ML engineers I’ve talked to that have no training in statistics AT ALL. That seems absurd to me. Wondering if it’s anecdotal or an actual thing?" from @bellmar" height="281" width="500">
I wasn’t expecting to get a response at all, certainly not a bunch of machine learning experts – some of whom I don’t know personally even – jumping in to say ….basically yes.
And in some cases "Yes, but who cares?"
The term "script kiddie" describes inexperienced technologists (historically hackers) who copy and paste code that someone else wrote and that they do not understand. Despite its wide use as a pejorative, the script kiddie mentality is how many of us began in software, downloading code, making changes to it and observing the results. We learned what code did through rigorous exploration of trial and error. If they don’t lose interest, script kiddies eventually grow up to become high quality software engineers.
In trying to build effective tooling for AI and machine learning engineers, we have accidentally endorsed applying a script kiddie mentality here too. Infrastructure and applications that allow developers to drop in pre-trained models are everywhere. How do you know whether the AI infused product you’re relying on was built by a practitioner who deeply understands the model and the math behind it and not someone copying and pasting code from Stack Overflow? You don’t.
Software is a field that has blossomed by embracing non-traditional backgrounds, re-skilled workers, and autodidacts. I myself have no formal training in computer science, my first forays into programming were script kiddie in nature – I’m not judging it. But despite the fact that AI is written in software, it is different from traditional software in that the line between something appearing to work correctly and actually working correctly is much broader. You can’t just eyeball results and deduce that a machine learning model is unbiased or bug free.
The State Graph of Safety
Those of you who follow my writing will know I’ve been thinking a lot about safety science and how it might fit (or not) into the development of AI. There’s a lot of talk, particularly in the defense industry, around how to regulate AI to ensure it’s responsible and ethical. How do you control the development and usage of algorithms? And who do you hold accountable for bad decisions made at the direction or with assistance of AI? How do we ensure this technology is safe?
What’s curious is that the policy conversation here fixates on a goal that safety scientists will tell you is impossible: we will never be able to produce AI technology that cannot cause injury.
But on the other hand… it’s impossible to produce anything that meets that standard. AI, power tools, children’s toys … anything that can be used can be misused too.
A technology isn’t considered safe because it is harmless. A technology is considered safe because the operator can accurately assess and mitigate the risks of using it. Cars aren’t safe because it’s impossible to ram one into a tree. Medical devices aren’t safe because it’s impossible to hurt someone with them. We’re confident in the safety of electrical sockets without insisting they have some kind of smart feature that will prevent us from sticking forks in them.
We consider a technology safe when we feel confident that we can predict how it will react to our interaction with it. Instead of regulating AI to prevent negative outcomes, we should be looking to manage the possible states between the operator and the technology. It’s useful to think of this as a graph where actions and contextual factors create networks of actions and reactions. We make a technology "safer" by limiting this state graph – that is limiting the number of things that could shift the operator’s ability to predict the system’s response.
One way we can shrink the state graph is by outlawing some states. We require cars to have brakes, seat belts and airbags because these features make certain risks of driving less likely. Fewer risks overall make it easier for the operator to feel confident that they understand the total risks of operating a vehicle and thus cars becomes safer.
But the other way we shrink the state graph is by raising the competency of the operator. When an operator can be anyone, of any age, at any experience level the set of behaviors we might see when the operator interacts with a piece of technology is impossibly large. When we can guarantee that the operator has a certain level of experience and has mastered certain background knowledge we eliminate many more edges of our state graph.
In other words, cars aren’t safe because it’s impossible to crash them. We require that cars come with seat belts, and that drivers wear them. But we also require that drivers have a license with a minimum amount of driving experience and a test of baseline knowledge.
However, when it comes to the issue of making AI safe we’re left with the question: Who is the operator in the first place? Is the operator the machine learning engineer who builds and trains the model? How much experience does that person have? What baseline knowledge must all AI engineers prove mastery over? What are the relevant degrees or credentials?
Or is the operator the downstream user? Should use of tools with AI integrations be restricted to people with specific training?
We wouldn’t let someone who hadn’t been certified in civic engineering build a bridge and we wouldn’t let someone without a license to practice medicine perform a surgery. Now in the mists of major breakthroughs in Artificial Intelligence, we are acting like there’s some magical policy fix that will eliminate all negative outcomes with AI while still leaving it open for anyone to build and for anyone to use.
Maybe that’s a bad idea. Are AI practitioners really less important than doctors, lawyers, real estate agents, or any other role we require people to be licensed to do?