How to define fairness to detect and prevent discriminatory outcomes in Machine Learning

In order to detect discriminatory outcomes in Machine Learning predictions, we need to compare how well our model treats different user segments.

This can be achieved is by defining a metric that describes the notion of fairness in our model. For example, when looking at university admissions, we can compare admission rates of men and women. This corresponds to the use of Demographic Parity as the mathematical definition of fairness. It states that in an ideal world, the difference between the groups getting the positive outcome (e.g.: getting admitted at university) should be zero. However, this rarely happens and it is why we need to track the gap.

Demographic Parity is one of the definitions we can use to describe mathematical fairness. In this post, we’ll review the three most common definitions that can be applied in classification models. It very important to select the right type of fairness; otherwise, the wrong metric can lead to harmful decisions that we will explore.

Getting started with an example

Throughout this post, I will be using an example of a credit model to explain the major definitions of mathematical fairness. We will look into when and how it makes sense to use them.

The historical data we’ll use can be described as follows:

• Paid loan: Those who paid in full their loans.
• Defaulted: Those who were not able to pay back their loan and defaulted.

Figure 1: example of a classification credit model

Let’s assume we’re building a credit model based purely on “income”. Our model will aim to learn the typical income of those who can pay their loan in full and differentiate them from others who tend to default.

That means that we simply need to set an income threshold in our training set to decide who gets a loan in the future (this is the dotted line in Figure 1). Those that are above the threshold, will get the loan (positive predictions). Those below the threshold are the ones who will not get a loan (negative predictions).

First Example of Mathematical Fairness: Demographic Parity

Demographic Parity states that the proportion of each segment of a protected class (e.g. gender) should receive the positive outcome at equal rates. A positive outcome is the preferred decision, such as “getting to university”, “getting a loan” or “being shown the ad”. As mentioned earlier, the difference should be ideally zero, but this is usually not the case.

Let’s assume we’re building a credit model based purely on “income” (see Figure 2). We may decide to use different requirement levels for each group so that the percentage of people getting a loan in Group A is equal to the percentage of people getting a loan in Group B.

Mathematical definition

In mathematical terms, Demographic Parity requires the outcome to be independent of the protected class A:

Equation 1: Probability of getting an outcome is independent of A

With regard to the confusion matrix, we require the Positive Rate (PR) to be the same for each segment of the protected class.

Positive Rate (A=0) = Positive Rate (A=1)

In practise, we may not require for the difference in Positive Rates to equal zero, but we will aim to minimise this gap.

What could go wrong?

Let’s assume we’d like to create a dancing group. We’re trying to select our dancing crew from a group of professional dancers, pop singers and a few actors. Historical stereotypes tell us that dancers and pop singers will be great at this job. However, we’re nearly certain that actors probably won’t.

Assuming we get an outcome as in the GIF above, professional dancers and pop singers will be perceived in this case as naturally superior. The bad dancing performance from Mr Bean will indeed reinforce my bias towards actors being bad at dancing.

By the time that the decision is made, actors have not had the same opportunity to learn to dance as professional dancers and pop singers. If we immediately ask for Demographic Parity, this can lead to promotion of historical biases.

Getting now to a more serious example. Let’s say we want to bring more women to a board composed of the typical “boys’ club”. This may not be the most welcoming environment for women. We need to have a plan in place so that the environment is inclusive and the new minority gets the right support. Otherwise, we risk reinforcing historical bias.

When to use Demographic Parity

We should use Demographic Parity as a definition of fairness when:

• We want to change the state of our current world to improve it (e.g.: we want to see more minority groups getting to the top)
• We are aware of historical biases may have affected the quality of our data (e.g.: ML solution trained to hire software engineers, where nearly no women was hired before)
• We have a plan in place to support the unprivileged group and to prevent the reinforcement of historical biases (e.g.: setting policies that penalise non inclusive behaviour at a board)

University admissions tend to work towards achieving Demographic Parity. A good example is Oxford University aiming to improve diversity by admitting more students from disadvantaged backgrounds. In this case, students have high academic potential, but their education might have been disrupted or they have had to overcome personal disadvantage. Students in this group will receive extra support before beginning their degree courses, extending to a complete foundation year.

Second Example of Mathematical Fairness: Equal Opportunity

Figure 3: use of different income thresholds to achieve Equal Opportunity

Equal Opportunity states that each group should get the positive outcome at equal rates, assuming that people in this group qualify for it.

If we look at our example in Figure 3, we can see the percentage of positives that were accurately predicted is 50% for both groups.

If we think about it, this is like saying:

“You’re allowed to be part of our dancing group, assuming you can dance”.

Mathematical definition

Equal Opportunity requires the positive outcome to be independent of the protected class A, conditional on Y being an actual positive:

Equation 2: Probability of getting a positive outcome is independent of A, conditional on Y being an actual positive

Based on the confusion matrix, we require the True Positive Rate (TPR) to be the same for each segment of the protected class.

True Positive Rate (A=0) = True Positive Rate (A=1)

In practise, we may not require for the difference in True Positive Rates to equal zero, but we will aim to minimise the gap.

What could go wrong?

Figure 4: many False Positives are introduced in Group B to be able to achieve Equal Opportunity

If we look at Figure 4, we can see that the True Positive Rate (TRP) is the same for both groups as required by Equal Opportunity.

However, if we pay attention to Group B, we can notice that many False Positives were introduced. False Positives in this case are those who get a loan when in fact they’re likely to default. That means that we would be damaging the credit score at a larger scale in Group B, which results in disparate impact.

When to use Equal Opportunity

We should use Equal Opportunity as a definition of fairness when:

• There is a strong emphasis on predicting the positive outcome correctly (e.g.: we need to be very good at detecting a fraudulent transaction)
• Introducing False Positives are not costly to the user nor the company (e.g.: wrongly notifying a customer about fraudulent activity will not be necessarily expensive to the customer nor the bank sending the alert)
• The target variable is not considered subjective (e.g.: labelling who is a ‘good’ employee is prompt to bias and hence very subjective)

As mentioned above, using Equal Opportunity may be a suitable case for fraud detection, given that mistakes in form of ‘False Positives’ do not generate a costly negative impact on customers or the company handling the situation. Also, labelling a transaction as fraudulent should be a clear decision, not subject to major bias.

Third Example of Mathematical Fairness: Equalised Odds

Figure 5: use of different income thresholds to achieve equalised odds

The final definition of mathematical fairness is Equalised Odds and it is the most restrictive concept from the ones we’ll discuss in this blog.

This concept states that the model should:

• correctly identify the positive outcome at equal rates across groups (same as in Equal Opportunity), but also
• miss-classify the positive outcome at equal rates across groups (creating the same proportion of False Positives across groups)

We can see such an example in Figure 5, where we are correctly identifying the positive outcome at a rate of 50% for both groups, but also having the same proportion of False Positives at 25% for both groups.

Mathematical definition

Equalised Odds requires the positive outcome to be independent of the protected class A, conditional on the actual Y:

Equation 3: Probability of getting a positive outcome is independent of A, conditional on Y

Based on the confusion matrix, we require the True Positive Rate (TPR) and False Positive Rate (FPR) to be the same for each segment of the protected class.

True Positive Rate (A=0) = True Positive Rate (A=1) , and

False Positive Rate (A=0) = False Positive Rate (A=1)

In practise, we may not require for the difference in True Positive Rates and False Positive Rates to equal zero, but we will aim to minimise both gaps.

What could go wrong?

Figure 6: The False Positive Rate (bar chart, left y-axis) is minimised when enforcing Equalised Odds at the expense of lower profits (line chart, right axis)

Given that this is the most restrictive of the definitions, trying to achieve equal TPR and FPR for each group may lead to a drop in profits. This is because model performance could be compromised by not being able to optimise accuracy on the majority group.

Figure 6 is an example of what could happen when Equalised Odds is enforced. I built a credit model to identify who should get a loan and analysed definitions of fairness for people over 40 and under 40. In this case, profits were heavily penalised during Equalised Odds compared to other definitions of fairness.

For this example, no work was done to detect or mitigate bias before building the model, which may explain why profits were so low. This also shows the importance to apply bias mitigation techniques such as “re-weighing” and to perform a good analysis of missing data to build a good model for everyone without compromising monetary value.

When to use Equalised Odds

We should use Equalised Odds as a definition of fairness when:

• There is a strong emphasis on predicting the positive outcome correctly (e.g.: correctly identifying who should get a loan drives profits), and
• We strongly care about minimising costly False Positives (e.g.: reducing the grant of loans to people who would not be able to pay back )
• The target variable is not considered subjective (e.g.: labeling who is a ‘good’ employee is prompt to bias and hence very subjective)
• The reward function of the model is not heavily compromised (e.g.: revenue or profit function for the business remains high)

Credit models are an excellent example to use Equalised Odds as a definition of fairness. We generate business value by accepting as many clients who could pay back a loan in full. However, we also care about minimising ‘False Positives’, those who are granted a loan and cannot pay back.

False positives can damage the credit scores of clients, and consequently their financial opportunities in the future. It would also incur a high cost to the bank generating the loan.

At the same time, labelling a loan as paid in full or defaulted is a non subjective decision and clear process.

Conclusion

In this blog, we learned the three most common definitions of ‘Mathematical Fairness’ that can be applied in classification models. We looked into when it makes sense to use each fairness metric and what to consider when applying them.

We must start analysing how our models may have a disparate impact on different customer segments. Otherwise, we risk propagating systematic discrimination at scale.

If you have any feedback or recommendations to expand this article, please leave them in the comment box below 👇.