Bias in predictive models — part 1/2
The first step in fighting bias is to define it.
Abstract
AI is already playing a significant role in our lives and soon it’s going to make more and more life-changing decisions. The problem of AI fairness and biases is attracting growing attention from both research and legislators. There exist a few possible definitions of what is bias and how to measure it, each having its own merits, but also — limitations of applicability.
Which one to use is an important (and not very simple) choice, that companies and, possibly, regulators would need to make.
Introduction
Remember this recent story about Amazon’s AI hiring tool which developed a bias against women? Fortunately, their team was able to identify the problem before the model went into production. AI is already playing a significant role in our lives and soon it’s going to make more and more life-changing decisions. Would you get that job offer? Get approved for that loan? Would your product be recommended? Would your social media account be blocked? Would your child get admitted to that university? Will you be “randomly” chosen for a deep tax audit? And even, will you get an early release from the jail? Not all decisions on this list are already done by AI, but many are, and this list is only going to grow every year.
In this ProPublica analysis on recidivism rates in Broward County, it was shown (see table below) that African Americans that didn’t re-offend, are almost twice as likely as whites to be (incorrectly) labeled as higher risk.
Most kinds of AI, if left to their own devices, would perpetuate whatever trends (including biases) that were present in the historical data they were trained on. If women were previously a small part of the workforce that is what the AI would learn (as in Amazon hiring tool story).
With this much at stake, it’s no wonder that the topic is getting more attention, both from researchers and now — also from legislators.
Figures are taken from this article
The issue poses two high-level problems:
- What is bias and how to measure it?
- If the bias is there, what can be done about it?
Both questions are not easy to answer. In this post, we concentrate on the definition of bias and ways to measure it. The ways of solving the issue will be discussed in a follow-up.
Definition of Bias
In an ideal world, everyone would want their models to be “fair”, and none would want them to be “biased” but what does it mean, exactly? Unfortunately, there seems to be no universal and agreed definition of bias or fairness.
In this great article about different ways to measure model fairness, David Weinberger, writer-in-residence at People + AI Research (PAIR) at Google, presents not less than five (!) different ways to define what is fair, each one with its own merits and drawbacks.
It’s interesting to note that some of the more intuitive methods, such as “group unaware” or “demographic parity” sometimes turn out as not suitable for real-life problems.
“Group unaware” method states that if one totally disregards protected class information (such as gender), the results would reflect the objective merits of applicants which is fair. If, as a result, there would be a disproportional representation of some gender — this is “what data says”. However, in real life, it’s often more complex. Gender information may be hidden in some other variables (proxies), and lower representation of women stemming from possible past discrimination would then creep into our new model (as happened in the Amazon case or this Google Translate example). There are problems like medical diagnosis where chances to have some condition may directly depend on sex. In other problems, the meaning of some parameters may depend on gender indirectly. For example, in CV analysis, certain gaps in employment for women may indicate maternity leave which has different meaning compared to employment gaps when a person couldn’t or wouldn’t get a job. In summary, sometimes it’s ok, or even desirable to refer to protected variables, and not referring to them does not guarantee that there would be no bias.
Another common method, “Demographic Parity” effectively postulates equality of outcome. Under this postulate, if there were 30% orange applicants, we expect to see 30% orange in approvals. The concept of “Demographic Parity” is very similar to “Disparate Impact” criteria, which is used in the US anti-discrimination legal framework that regulates areas such as Credit, Education, Employment, etc. However, the application of this method is far from obvious. Were these orange applicants as qualified as blue ones? What should be taken as 100%? Some methods to mitigate the disparity can lead to situations where similar people are treated differently, which by itself can be seen as discriminatory. One such case even reached the Supreme Court and the general topic is the subject of legal discussion which is beyond the scope of this overview.
This assumption of equal outcome may be right in some cases, but is it always correct? Sometimes, the protected variable has a correlation with the output for reasons which have nothing to do with bias. For example, the rate of occurrence of certain medical conditions may vary a lot between different genders or ethnicities.
Recently, Intel declared that they reached “full gender representation” with only about 27% female staffers. The naive application of the “Demographic Parity” would, of course, fail because 27% is far less than the rate of females in the general population. Intel could make this claim because they used as reference only females that were the graduates of relevant college degrees, as only them, according to Intel, constitute the pool of qualified female candidates.
In statistics, the bias of an estimator is defined as the difference between the estimator’s expected value and the true value of the parameter being estimated. However, in real life, it’s often hard to know what this “true value” is. If that declined borrower had been approved for the loan, would they actually have repaid it?
Sometimes we have the luxury of knowing the actual outcome for all candidates, including declined ones. For example, in this case, the model was predicting the likelihood of existing offenders committing a future crime, and a couple of years after their release, we can know who actually committed another crime and who didn’t. In the loan case, people who were declined loan by a particular bank might still have been able to get a loan elsewhere, and credit bureau may have information on their subsequent payments. For such cases, where the actual outcome is known, the following criteria, proposed by Moritz Hardt from Google Brain Team and teacher of “Fairness in Machine Learning” at UC Berkeley, seems to be applicable:
the system ought to be tuned so that the percentage of times it’s wrong in the total of approvals and denials is the same for both classes
In other words, we define as biased (and want to avoid) the situations where approval of bad applicants or decline of good applicants differ too much for different classes (e.g. genders). Note, we do not need to assume anything about the rate of correct approvals or declines. Appealing as it is, this metric is somewhat less intuitive and, of course, is only applicable when actual outcomes are known e.g. in backtesting on historical data.
To summarize, there are multiple ways to define and measure bias, which may suit more or less for your particular problem. Which one to use is an important (and not very simple) choice, that companies and, possibly, regulators would need to make.
After some way of detecting bias is chosen, the next challenge is to mitigate it. Some ways to mitigate the bias, both existing and novel ones, will be discussed in the next post.
