PODCAST
Catching edge cases in AI
Yaron Singer on building more robust, fault-tolerant AI systems
Editor’s note: The TDS Podcast is hosted by Jeremie Harris, who is the co-founder of Mercurius, an AI safety startup. Every week, Jeremie chats with researchers and business leaders at the forefront of the field to unpack the most pressing questions around data science, machine learning, and AI.
It’s no secret that AI systems are being used in more and more high-stakes applications. As AI eats the world, it’s becoming critical to ensure that AI systems behave robustly — that they don’t get thrown off by unusual inputs, and start spitting out harmful predictions or recommending dangerous courses of action. If we’re going to have AI drive us to work, or decide who gets bank loans and who doesn’t, we’d better be confident that our AI systems aren’t going to fail because of a freak blizzard, or because some intern missed a minus sign.
We’re now past the point where companies can afford to treat AI development like a glorified Kaggle competition, in which the only thing that matters is how well models perform on a testing set. AI-powered screw-ups aren’t always life-or-death issues, but they can harm real users, and cause brand damage to companies that don’t anticipate them.
Fortunately, AI risk is starting to get more attention these days, and new companies — like Robust Intelligence — are stepping up to develop strategies that anticipate AI failures, and mitigate their effects. Joining me for this episode of the podcast was Yaron Singer, a former Googler, professor of computer science and applied math at Harvard, and now CEO and co-founder of Robust Intelligence. Yaron has the rare combination of theoretical and engineering expertise required to understand what AI risk is, and the product intuition to know how to integrate that understanding into solutions that can help developers and companies deal with AI risk.
Here were some of my favourite take-homes from the conversation:
- Unlike traditional software systems, errors in AI systems tend to be silent. A bad prediction or faulty data input doesn’t usually lead to a crash, or even an error warning. That creates a challenge for AI developers, who then have to find clever ways to anticipate and detect a more subtle class of errors than those software engineers might encounter.
- Software engineering best practices usually involve test-driven development (TDD), a procedure in which developers start by writing tests that their code will have to pass, before writing the code itself. Yaron argues that a similar approach is becoming necessary in AI: like software engineers in the 2000s, AI developers today waste large amounts of time tracking down (often silent) errors and debugging their models — both of which become less necessary when developers use robust testing protocols.
- Yaron points to many sources of AI risk and model failure. Out-of-distribution sampling, for example, occurs when a model encounters inputs in the wild that differ from those it was trained or tested on, and often leads to errors. But errors in data entry, or even outright malicious attacks that leverage adversarial AI, are also important sources of AI failure risk.
- Data preprocessing is often the most vulnerable stage of the data lifecycle when it comes to AI risk. Data that’s obviously faulty (e.g. wrong data type) can be detected and flagged fairly easily, but more challenging model failures happen because data preprocessing lacks context about what inputs are reasonable or expected. For example, a user might enter their birth year instead of their age, and throw off a preprocessing scheme that isn’t designed with the context awareness needed to recognize that “1985” lies outside the appropriate range of age values. Dealing with these failure classes can involve designing good tests, or even developing anomaly detection models that catch and flag outlier input values.
- One concept that Yaron borrows from the cybersecurity world is that of fuzzing. A fuzzer is a tool that generates (usually random) inputs that can be fed to a program in the hopes of making it break, thereby revealing vulnerabilities it might contain. In a sense, Robust Intelligence is doing just that with their clients’ machine learning models: they’re in the business of building fuzzers that can cause AI systems to generate nonsense outputs.
Chapters:
- 0:00 Intro
- 2:30 Journey into AI risk
- 5:20 Guarantees of AI systems
- 11:00 Testing as a solution
- 15:20 Generality and software versus custom work
- 18:55 Consistency across model types
- 24:40 Different model failures
- 30:25 Levels of responsibility
- 35:00 Wrap-up
