Inside AI

A primer on AI fairness

What it is and the tradeoffs to be made

Photo by Bill Oxford on Unsplash

1. Purpose

This is a primer for what is AI, what is AI fairness, why fairness is important, how bias creeps up into the system, how to tackle algorithmic bias, and the profit tradeoff. This is a broad and complex topic. So to narrow down the scope, this article is not about:

• artificial general nor super intelligence. It is about the products and services that we are already using
• killer robots nor autonomous weapons
• intentional harm but rather unintentional bias
• data privacy, misinformation, nor antitrust. Although fairness is related to these
• engendering societal trust in AI, which includes issues of transparency, explainability, and accountability
• how businesses should hire and manage teams to address bias

Note on terms: Algorithms, artificial intelligence, automated decision making systems (ADMs), machine learning, and models are used interchangeably in this paper.

2. What is Artificial Intelligence

Artificial intelligence means different things to different parties. The diagram below helps delineate the differences. Computer scientists would define AI broadly as techniques that mimic human behavior, this includes if-then programs, knowledge bases, and machine learning among others. Corporations typically meant machine learning, a subset of AI. Machine learning (ML) is a computer program that learns from data to perform a particular task (e.g. prediction, classification). ML includes algorithms such as regression, random forests, and deep learning among others.

Advances in artificial intelligence discussed in media today typically refers to advances in machine learning, particularly deep neural networks Neural networks loosely mimic how the brain works by modeling each interconnected neuron as a mathematical function receiving inputs and produces an output. As more neurons are added, the network increases its ability to model more complex relationships and increases model accuracy [1]. To oversimplify it, think of neurons as building blocks of our brains. More neurons = smarter (potentially).

3. Defining bias and fairness

In statistics, bias means an inaccurate representation of what is true (e.g. the underlying population). In a social context, bias is a preference for an outcome or group of people, whether it is fair and unfair. Colloquially and in this article, bias means unfair or unwanted bias.

There is no one definition of what is fair. What is considered fair depends on the context. This challenge also presents a need and opportunity for people to critically think, explicitly express, and quantify what is considered fair. Decades ago, it was considered “normal” for “blacks” to be slaves in the United States. During the Spanish occupation of the Philippines, Chinese (like me) who were forced to live in the outside cities because of their race. Now, discriminating based on skin color or race is both illegal and socially unacceptable.

3.1. How can bias and fairness be defined?

3.1.1. Group vs individual

One way to frame fairness is by group or individual fairness. Group fairness requires that different groups, defined by specific attributes, should have similar outcomes. Group attributes are usually defined by law in practice but it could be anything — “protected attributes”. The most common attributes are gender and race. See Appendix 7.1 for a list of laws and regulations that define protected attributes. Laws and regulations make it easier to measure group fairness.

Individual fairness requires that similar individuals should have similar outcomes. This makes intuitive sense but is difficult to put into practice. How to establish similarity between individuals differs by context and is difficult to measure. The credit score of two individuals matter more when evaluating mortgage applications but less so in evaluating MBA applications. It is impossible to set a standard definition of individual fairness that applies to all contexts.

3.1.2. Example of specific metrics and trade-offs

As an example, we could imagine a situation where a bank is evaluating 100 mortgage applications. 70 are from men and 30 are from women [2]. The protected attribute is gender. One measure of fairness is demographic parity. The percentage of men and women who get approved should be the same. If 35 men (50%) get their mortgage approved, there should be 15 for women (50%). However, in some sense this is unfair if there is a higher percentage of loan-worthy individuals in either group. Some argue that what matters more is that the model’s prediction accuracy should be the same for each group. This is a measure of equal opportunity or accuracy. If 90% of men whose mortgage were approved are expected to pay back the loan, then it should also be 90% for women. Both definitions of fairness make sense and will likely conflict with each other. A hands-off approach to fairness is by intentionally excluding the protected attribute and any closely correlated variables (which is difficult to do in practice) in making decisions. This is called group unaware.

Appendix 7.2 shows a non-exhaustive list of 19 fairness metrics. Some researchers have proven that some definitions conflict with each other. Understanding the tradeoffs are important because it directly impacts us and business profits.

4. Why should we be concerned

4.1. Societal impact

Cathy O’ Neil’s award-winning book “Weapons of Math Destruction” showed how ADMs are harming the lives of people in getting a job, loan, insurance, and fair justice [3]. ProPublica studied how a software system used by some US courts will predict that “blacks are almost twice as likely as whites to be labeled a higher risk but not actually re-offend.” [4] More blacks might have incorrectly stayed in jail longer which marginally increases recidivism [5]. This is evidence of a vicious feedback loop. Over 90% of companies use an automated resume screener [6] and Amazon’s scrapped automated recruiting tool was biased against women [7]. These examples are not meant to show that ADMs are inherently dangerous but rather how embedded it is in our society. ADMs are tools that are usually built with good intentions and have positive benefits.

ADMs are often built to make consistent and accurate decisions at scale. Humans tend to make inconsistent and biased decisions. Judges were shown to reduce their likelihood of favorable rulings to 0% right before lunch or break before returning to an average of 65% otherwise. Other examples can be found in Appendix 7.3.

4.2. Business impact

ADMs have also allowed the influx of web-scale demands of internet access. Each job opening receives 250 applications. Google receives 3 million resumes a year and hired 7,000. It is much more efficient to automatically screen candidates. it frees up recruiters to build personal connections and interview candidates. This increases the productivity of each recruiter (i.e. lowers cost for a business).

Going back to the example of mortgage loans in section 3.1.2, researchers have developed a tool that shows how different fairness definitions lead to different profits [8]. The hypothetical example presented how one fairness definitions results in profits 21% lower than the theoretical maximum. Section 6 discusses the profit trade off more.

5. Where bias comes from and how to tackle it

The diagram below shows where bias creeps into the machine learning process and where interventions could be made [9]. The use of ‘creeps in’ is intentional. Most of developers do not intentionally build bias into the models.

5.1. Sources of bias

Bias can appear because of the data, algorithm, and how the model is used. The data used to train the model can either be unrepresentative/incomplete or already biased. Unrepresentative training data is by definition biased. The model will perform poorly, regardless of whether it is used to discriminate against a certain group or individual. Unrepresentative dataset could arise for various reasons such as incomplete data collection and oversampling a subpopulation. Biased datasets are data recording past biased decisions. Models trained on this data will simply reflect and reinforce the bias.

The model building process involves selecting what features to include in the model and what algorithm to use. In most cases, the model builder (e.g. data scientist, machine learning engineer) selects what features will be included in the training dataset. What features get selected will incorporate biases of the builder. In most algorithms, there is a cost function that is being minimized. In the case of linear regression, it is the squared-deviations. The cost function is usually set to maximize accuracy but it can be configured to optimize different metrics. However, usually it is set to maximize accuracy which often reinforces bias patterns embedded in the dataset.

How the model is used and evaluated also lead to bias. In section 3.1.2’s loan example, the threshold used to determine whether to approve a mortgage is set by the user. Different thresholds reflect different fairness measurements. The performance of models are also evaluated by using different metrics, often accuracy is used. The use of fairness metrics, or the lack of, also leads to bias.

5.2. Bias mitigation methods

Bias mitigating methods are categorized by the stage of intervention during the machine learning pipeline. Fair pre-processing methods tackle biases present in data. Fair in-processing methods modifies the learning process to ‘disregard’ sensitive attributes or penalize algorithms for being ‘unfair’. Fair post-processing methods modifies an existing models’ outputs. The diagram below is a good overview of the more common methods [10]. We will not dive into each specific method since it will be too technical. One pre-processing method that is not listed below is collecting more (useful) data.

6. Profit vs. fairness tradeoff

The issue for businesses right now are awareness, uncertainty about standards, lack of qualified methods to ability to solve biases [11]. This is where most efforts are being poured into. There are several working groups and institutes that are leading the charge to define fairness and educate both private and public constituents (e.g. AI NOW, OpenAI).

However, one issue that will become more prominent as the former are addressed is how it will impact profits. Profits motivate firms, amidst regulatory, moral, and societal constraints. As section 4.2 illustrates, there is a tension between profits and fairness. Firms are not naturally incentivized to ensure fair machine learning algorithms. One way to think about how predictions link to profits is illustrated below with confusion matrices. A machine learning algorithms value is being able to increase the number of true positives and true negatives, which each have a value attached. Each false positive and false negative is costly. The value assigned to each depends on each context. A false negative is more costly in medical situations while a false positive is costlier in death penalty decisions. Expected value is profits that businesses can expect from using the algorithm. The more accurate the model, the higher the profits.

Bias mitigating methods that decrease accuracy will face resistance from businesses. Unfortunately, there is usually a tradeoff between accuracy and fairness. Different bias mitigating methods have different (negative) impact on accuracy. Pre-processing methods have the least impact on accuracy and hence are more valuable for businesses [12]. Collecting more data may increase accuracy further while also improving fairness. However, data collection and processing is expensive and the most time consuming part of the machine learning pipeline. This is one of practitioners’ biggest pain points, as surveyed by Wallach [12]. For example, there is no guideline or best practice on what additional data to collect, partly because each use case is requires different data. Being aware and clear about this trade off is important, because business leaders can make a more informed, and hopefully better, choice.

7. Appendix

7.1. List of Regulated Domains in the United States

Regulated domains

• Credit (Equal credit opportunity act)
• Education (Civil rights act of 1964; Education amendments of 1972)
• Employment (Civil rights act of 1964)
• Housing (Fair housing act)
• Public accommodation (Civil rights act of 1964)

Legally recognized ‘protected classes’

• Race (Civil rights act of 1964)
• Color (Civil rights act of 1964)
• Sex (Equal pay act of 1963, Civil rights act of 1964)
• Religion (Civil rights act of 1964)
• National origin (Civil rights act of 1964)
• Citizenship (Immigration reform and control act)
• Age (Age discrimination in employment act of 1967)
• Pregnancy (Pregnancy discrimination act)
• Familial status (Civil rights act of 1964)
• Disability status (Rehabilitation act of 1973, Americans with disabilities act of 1990)
• Veteran status (Vietnam era veteran’s readjustment assistance act of 1974, Uniformed services employment and reemployment rights act)
• Genetic information (Genetic information nondiscrimination act)

Discrimination Law: Disparate Treatment (formal vs intentional) vs Disparate Impact (20% rule, legal rule of thumb). Disparate treatment can be thought of as procedural fairness. The underlying philosophy is equality of opportunity. Disparate impact is distributive justice. There is tension between these two goals.

Source: Fairness in Machine Learning, NIPS 2017 Tutorial by Solon Barocas and Moritz Hardt (https://mrtz.org/nips17/#/)

7.2. Definitions of fairness

Source: Fairness and Machine Learning by Solon Barocas, Moritz Hardt, Arvind Narayanan (https://www.fairmlbook.org)

7.3. Additional examples of how AI/ADMs improves societal outcomes

Relative to human screeners, hiring algorithms yields candidates that are more diverse and likely to pass interviews, accept job offers, and perform better at work [13]. Using ADMs in consumer lending decisions will increase long-run profits while also reducing bias against older and immigrant borrowers [14]. Replacing judges with algorithms to predict recidivism increases social welfare. Crime can be reduced by up to 25% across all categories of crime, jail populations can be reduced by 42% while also significantly reducing the percentage of African-Americans and Hispanics in jail [15].

ADMs have also allowed the influx of web-scale demands of internet access. Each job opening receives 250 applications [16]. Google receives 3 million resumes a year and hired 7,000 [17]. It is much more efficient to automatically screen candidates. it frees up recruiters to build personal connections and interview candidates.

End notes

[1] McKinsey uses the same shorthand reference for its analysis of artificial intelligence use cases. There are several neural network techniques such as feed forward, generative adversarial networks, and convolutional neural networks. Refer to McKinsey’s study for a brief on the most common and valuable techniques in a business setting. From “Notes from the AI frontier: Applications and value of deep learning.” Accessed December 4, 2018. https://www.mckinsey.com/featured-insights/artificial-intelligence/notes-from-the-ai-frontier-applications-and-value-of-deep-learning.

[2] Adopted from Google PAIR’s What-if-Tool. https://pair-code.github.io/what-if-tool/ai-fairness.html

[3] Cathy O’ Neil. “Weapons of Math Destruction.” Broadway Books. 2016.

[4] ProPublica. “Machine Bias”. 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

[5] Bo Cowgill and Catherine Tucker. “Economics, Fairness and Algorithmic Bias”. In preparation for Journal of Economic Perspectives. 2019. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3361280

[6] Lauren Weber. “Your Resume vs Oblivion”. Wall Street Journal. 2012. https://www.wsj.com/articles/SB10001424052970204624204577178941034941330

[7] Jeffrey Dastin. “Amazon scraps secret AI recruiting tool that showed bias against women”. Reuters. 2018. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G

[8] Attacking discrimination with smarter machine learning. https://research.google.com/bigpicture/attacking-discrimination-in-ml/

[9] IBM. “AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias”. https://arxiv.org/abs/1810.01943

[10] It is from a blog but is the only great summary I could find. No paper summarizes existing methods as succinctly as this one. Ajitesh Kumar. “Machine Learning Models: Bias Mitigation Strategies”. https://dzone.com/articles/machine-learning-models-bias-mitigation-strategies

[11] A great survey of what issues industry practitioners face is done by a team of Microsoft Researchers. They interviewed 35 and surveyed 267 practitioners across a wide range of firms, encompassing different industries and products. Wallach, et. al. “Improving Fairness in Machine Learning Systems: What do Industry Practitioners Need?” https://arxiv.org/abs/1812.05239

[12] Roth, et. al. “A Comparative Study of Fairness-Enhancing Interventions in Machine Learning”. https://arxiv.org/abs/1802.04422

[13] Bo Cowgill. “Bias and Productivity in Humans and Algorithms: Theory and Evidence from Resume Screening”. http://conference.iza.org/conference_files/MacroEcon_2017/cowgill_b8981.pdf

[14] Will Dobbie, Andres Liberman, Daniel Paravisini, Vikram Pathania. “Measuring Bias in Consumer Lending”. NBER. 2018. https://www.nber.org/papers/w24953

[15] Jon Kleinberg, et al. “Human Decisions and Machine Predictions”. NBER. 2017. https://www.nber.org/papers/w23180

[16] John Sullivan. “ Why You Can’t Get A Job. Recruiting Explained By the Numbers”. ERE. 2013. https://www.ere.net/why-you-cant-get-a-job-recruiting-explained-by-the-numbers/

[17] Max Nisen. “Here’s why you only have a 0.2% chance of getting hired at Google“. Quartz. 2014. https://qz.com/285001/heres-why-you-only-have-a-0-2-chance-of-getting-hired-at-google/