Ethics in AI: Potential Root Causes for Biased Algorithms
An alternative approach to understanding bias in data
A s the number of data science applications increases, so does the potential for abuse. It is easy to condemn developers or analytics teams for the results of their algorithms, but are they the main culprits? The following article tries to discuss the problem from a different angle and concludes that ethical abuse might be the real problem with data in our society.
A better world won’t come about simply because we use data; data has its dark underside. ¹
Biased algorithms
Today, discussions of data science are often associated with significant opportunities in business and industry to deliver more accurate predictive and classification solutions and improve people’s lives in the health and environmental sectors.² ³ Perhaps as important as the benefits are the data ethics challenges that should be considered when evaluating new solutions and approaches in data analytics. As a specific branch of ethics, data ethics addresses moral issues related to data, algorithms, and the respective practices to formulate and support morally good solutions.⁴ Overall, there seems to be a fine line between the use and misuse of data. It is common for companies to collect data not only to increase their profits but also to provide a tailored and targeted experience to their customers.⁵ However, when companies start to externally exploit their data for purposes other than those for which it was collected, ethical questions arise. In addition to these privacy-related issues, another challenge is data analysis bias, which can arise in several ways. Examples include the creation of survey questions by people with a particular intent or framing, the selective collection of data from groups with a particular background, or the underlying bias of those from whom the data is drawn.⁶
Popular examples of algorithmic discrimination
The issues are not at all theoretical — they lead to real concerns and cause serious discussions as described in several case studies:
- Racial discrimination in crime prediction
A problematic and well-known example is the prediction of crime. A 2016 machine learning paper by Shanghai Jiao Tong University investigated whether crime can be detected in people based on facial feature analysis.⁷ The result was that members of the population who look “different” from the average are more likely to be suspected of committing crimes. ⁸ Another example of biased data science approaches is the risk assessment algorithm used in the United States to predict the likelihood of re-offending (repeated criminal behavior) by arrested individuals. It turned out that the underlying algorithm tended to predict the likelihood of recidivism of black defendants twice as high as that of white defendants, while white defendants tended to be incorrectly classified as low-risk when in fact they were recidivists.⁹ - Gender discrimination in recruitment
Another area that is affected by biased algorithms is recruitment. In 2014, Amazon used machine learning-based software to evaluate and rank the CVs of potential candidates to find new top talent. In 2015, the company discovered that its new system was not evaluating applicants in a gender-neutral way, especially for software development jobs. Based on hiring over the last ten years, the system penalized all applications that included the word “women” in their CVs.¹⁰ - Stereotypes in the social media
Gender and race bias can also be found in Facebook’s advertising. In their 2019 empirical study, Ali et al. discovered that there may be biased rules for delivering ads to specific users, based on the predicted relevance of those ads to different audiences. This relevance is often based on male/female stereotypes, classified and correlated through image analysis software that analyses digital advertising media.¹¹
Principle for big data ethics
How can this be prevented? These cases are just a few of many instances where algorithms are used in a biased way in today’s world. Many experts discuss and evaluate ways to avoid bias in data analysis, which is expressed in one of the “5 Principles for Big Data Ethics” published by Herdiana: “Big Data should not institutionalize unfair biases such as racism or sexism.” ¹² While this principle is considered a general rule within data ethics, the examples given “such as racism or sexism” seem to be the most common occurrences. However, a more general description might be:
“Big data should not institutionalize unfair biases that cause discrimination”.
The Canadian Human Rights Commission (CHRC) describes discrimination as an act or decision that treats a person or group poorly for reasons called grounds. These grounds are protected by the Canadian Human Rights Act and include race, national or ethnic origin, color, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, disability, genetic characteristics, and a conviction for which a pardon has been granted or registration suspended. ¹³ In sociology, discrimination describes the belittling of others based on individual or group characteristics, practiced systemically and therefore constituting a gross violation of human rights.¹⁴ The European Convention on Human Rights (ECHR) also prohibits discrimination in all of the above areas in Article 14, adding to the CHRC list language, political opinion, and property as well.¹⁵
The European Economic and Social Committee (EESC) study entitled “The ethics of Big Data: Balancing economic benefits and ethical questions of Big Data in the EU policy context” describes algorithm bias as one of the critical ethical problems of Big Data. They define trust in algorithms as an ethical problem because most people think that machines are neutral by definition. In reality, this is not the case, and therefore the risk can be very high.¹⁶ But what are possible reasons for this lack of neutrality? It is hard to imagine that the best data scientists in the world have developed their algorithms with this goal in mind.
Root causes analysis
Algorithms are not biased. Nevertheless, there exist many popular examples of skewed outcomes from algorithms while deeper assessments of the causes and drivers of the current outcome are often superficial. Moreover, the possible alternatives and outcomes in a hypothetical world without algorithms are missing. Therefore, the question of biased algorithms should not focus exclusively on the neutrality of algorithms, but also highlight the (missing) causes. The following analysis, therefore, aims to further explore issues related to bias and how today’s ethical pressure on developers could be understood as a collective responsibility. Nevertheless, it is crucial to underline the importance of eliminating discrimination in both technical and non-technical world, as the two are directly linked and influence each other:
- A biased society leads to biased data
By design, discrimination takes place before the data is even collected, as datasets are historic collections that have been influenced by society itself. To think about the technical aspects of this issue, one needs to examine the possible causes of biased systems. In most papers and studies, the authors state that the algorithms themselves are biased and cause discrimination.¹⁷ ¹⁸ However, it is difficult to ascribe the very cause of discrimination to algorithms themselves, since algorithms perform decision rules based on a given input and the interactions of given parameters in an automated way. ¹⁹ The rationale behind machine learning algorithms is usually an optimization problem where rules are applied to parameters based on their past effect on the outcome.²⁰ The presence of bias in the interaction between a given variable (e.g. gender) and the target (e.g. suitability for a vacancy) is therefore not caused by the algorithm itself, but by the underlying data set. Looking at the Amazon recruiting example, the biased outcome (i.e., favoring male applications vs. female) can be attributed to the training dataset initially fed to the algorithm. Assuming that the majority of former employees in technical positions were male, the algorithm anticipated this would make a success factor. So discrimination does not begin with the algorithm or its developer as an individual, but rather with the very hiring behavior of the company, which has been perpetrated for decades. - Hidden bias between variables
Provocatively, one could argue that every dataset is biased to some extent since it originated in a likely biased environment. An algorithm can only factor in what is present in the (biased) data, and data usually only contains what is present in the (biased) non-technical world. A reasonable and common-sense approach would be to remove such discriminatory variables from the dataset before beginning to model. However, no guarantee removing a variable will remove the presence or influence of that variable on the rest of the dataset. Let us imagine a fictitious dataset containing the variables gender and sport. Both variables could have an interaction, such as a “correlation between men and football and women and riding”, although this blunt example may not fully represent our society, since there may be women who prefer football to riding and vice versa. However, due to an unbalanced (e.g., male-heavy) dataset, it is likely that the algorithm indirectly favors candidates with a football preference over others, as the training process is based on biased information. This example shows that removing critical variables can reduce the risk of discrimination, but is not sufficient to prevent it. Moreover, it is of great importance how balanced the data sets represent the different expressions of a variable. A sensitive (discriminatory) item may be less problematic if the expressions are evenly distributed. If a subset of entries is underrepresented, the algorithm might factor this in negatively. Therefore, it is difficult to establish whether a dataset is truly unbiased.²¹ - Technical decisions vs. human decisions
There is no guarantee that humans make less biased decisions than algorithms. In addition to the idea that any dataset can be biased and lead to biased results in an algorithmic process, another key indicator of unfair predictions/classifications is the lack of information, in the form of incomplete data.²² It is difficult to guarantee that a given dataset possesses full visibility to explain a target behavior. Common reasons for the lack of data may be limited access, active exclusion of information, or lack of knowledge about the marginal impact of additional information. Another risk for discrimination does not only result from the technical implementation but can also be rooted in the existing structures of our society. The advantage of Big Data in combination with machine learning algorithms is the possibility to apply human rules and decisions to a large amount of information. If algorithms are considered biased, one might wonder how humans would have processed the analysis of given data. Looking back to social media advertising, personalized advertising aims to make individual suggestions based on one’s preferences. This implies that algorithms support individualism. However, algorithms have the opposite effect, as they process automated and standardized decision rules based on general rather than individual patterns. Ultimately, it can be assumed that algorithms support collectivism rather than individualism.²³
Conclusion
Our analysis has shown that the bias in algorithms is mainly a result of bias in today’s society. The latter supports the collection of biased datasets, which may contain (hidden) relationships, ultimately leading to biased results. Furthermore, we question whether human decision-making is less biased compared to algorithmic rules. Lastly, we have concluded that more complex algorithms are often too non-transparent.
Mitigation strategy
In discussions about the responsibility for biased results of machine learning algorithms, the developer or the data science team is often blamed for discriminatory behavior. On one hand, this argument is overly simplistic, and the apportionment of the blame needs to be further differentiated. On the other, any developer working with machine learning algorithms should be aware of the sensitivity of their task and their respective responsibilities. Since algorithms support collectivism, it is important to investigate the fairness of these collective algorithms that make decisions or suggestions for individuals. Literature suggests other techniques to reduce bias, such as a system design that randomizes a small sample of queries or the use of propensity weighting techniques.²⁴ For bias in human-based content, there are several ideas for quantifying discrimination and assessing the fairness of algorithms.²⁵
Fairness as a shared responsibility: both in the technical and real world
It is the responsibility of data teams to identify and minimize biased behavior in their data models. However, looking at the discriminatory factors examined in this article, it is a global responsibility to ensure that no one experiences such injustices, either through technical systems or human actions:
- Humans must treat humans equally and without discrimination
- Algorithms must treat humans equally and without discrimination
- We should not hide behind algorithms to justify biased and discriminatory human behavior
Outlook: now what?
It seems important to evaluate decisions or predictions not only based on the outcome itself but also based on the ethical considerations involved in working out the solution. We also need to rethink systems that involve biased behavior. However, there is no evidence that the alternative (human-based decision-making) leads to more equality in general. Consequently, we should not only focus on the problems with bias within technology but rather broaden the discussion to include bias outside technology. Nevertheless, precisely because of this, unbiased systems should help improve our current social weaknesses and possibly lead to more equality in the world to come. People could use AI to more easily detect discrimination in the non-technological world. Because wherever we find bias in the data, there is also bias in society.
I hope you find it interesting. Let me know your thoughts and feel free to connect on LinkedIn https://www.linkedin.com/in/jonas-dieckmann and/or to follow me here on medium.
See also some of my other articles:
References
[1]: Loukides, M., Mason, H., & Patil, D. (2018). Ethics and Data Science.
[2]: Floridi, L., & Taddeo, M. (2016). What is data ethics? Philosophical Transactions Of The Royal Society A: Mathematical, Physical, And Engineering Sciences, 374(2083), 20160360. doi: 10.1098/rsta.2016.0360
[3]: Google (2021). AI FOR SOCIAL GOOD, Applying AI to some of the world’s biggest challenges Retrieved 10 March 2021, from https://ai.google/social-good
[4] Mittelstadt, B., & Floridi, L. (2015). The Ethics of Big Data: Current and Foreseeable Issues in Biomedical Contexts. Science And Engineering Ethics, 22(2), 303–341. DOI: 10.1007/s11948–015- 9652–2
[5] Hand, D. (2018). Aspects of Data Ethics in a Changing World: Where Are We Now? Big Data, 6(3), 176–190. DOI: 10.1089/big.2018.0083
[6] Broad, E., Smith, A., & Wells, P. (2017). Helping organizations navigate ethical concerns in their data practices. Open Data Inst.
[7] Wu, Xiaolin & Zhang, Xi. (2016). Automated Inference on Criminality using Face Images.
[8] Sullivan, B. (2016). A New Program Judges If You’re a Criminal From Your Facial Features. Retrieved 10 March 2021, from https://www.vice.com/en/article/d7ykmw/new-programdecides-criminality-from-facial-features
[9] Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine Bias. Retrieved 28 February 2021, from https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
[10] Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Retrieved 28 February 2021, from https://www.reuters.com/article/us-amazon-com-jobsautomation-insight-idUSKCN1MK08G
[11] Ali, M., Sapiezynski, P., Bogen, M., Korolova, A., Mislove, A., & Rieke, A. (2019). Discrimination through Optimization. Proceedings Of The ACM On Human-Computer Interaction, 3(CSCW), 1–30. DOI: 10.1145/3359301
[12] Herdiana (2013), “5 Principles for Big Data Ethics,” Journal of Chemical Information and Modeling. Retrieved 10 March 2021, from: https://aiforceos.com/2018/06/16/big-data-ethics
[13] Canadian Human Rights Commission. What is Discrimination. Retrieved 28 February 2021, from https://www.chrc-ccdp.gc.ca/eng/content/what-discrimination
[14] Hartfiel, G., & Hillmann, K. (1994). Dictionary of Sociology. Stuttgart: Kröner
[15] Council of Europe (1950). European Convention for the Protection of Human Rights and Fundamental Freedoms. Retrieved 10 March 2021, from www.refworld.org/docid/3ae6b3b04
[16] European Economic and Social Committee. (2017). The ethics of Big Data.
[17] Eslami, M., Vaccaro, K., Karahalios, K. and Hamilton, K. (2017) ‘Be Careful; Things Can Be Worse than They Appear’: Understanding Biased Algorithms and Users’ Behavior Around Them in Rating Platforms. Proceedings of the International AAAI Conference on Web and Social Media, 11(1).
[18] Kim, P. (2017) Auditing Algorithms for Discrimination, 166 U. Pa. L. Rev. Retrieved 10 March 2021, from scholarship.law.upenn.edu/penn_law_review_online/vol166/iss1/10
[19] Mittelstadt, B., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2), 205395171667967. DOI: 10.1177/2053951716679679
[20] Lehr, D., Ohm, P. (2017). Playing with the Data: What Legal Scholars Should Learn About Machine Learning, UCDL Review 51: 653, 671.
[21] Calders, T., & Žliobaitė, I. (2013). Why Unbiased Computational Processes Can Lead to Discriminative Decision Procedures. Studies In Applied Philosophy, Epistemology, And Rational Ethics, 43–57. DOI: 10.1007/978–3–642–30487–3_3
[22] Rosen, R. (2019). Missing Data and Imputation. Retrieved 10 March 2021, from https://towardsdatascience.com/missing-data-and-imputation-89e9889268c8
[23] Saake. I., Nassehi, A. (2004). Die Kulturalisierung der Ethik. Eine zeitdiagnostische Anwendung des Luhmannschen Kulturbegriffs. Frankfurt/M: 102, 135
[24] Ridgeway, G., Kovalchik, S., Griffin, B., & Kabeto, M. (2015). Propensity Score Analysis with Survey Weighted Data. Journal Of Causal Inference, 3(2), 237–249. doi: 10.1515/jci-2014–0039
[25] Floridi, L. (2016). Faultless responsibility: on the nature and allocation of moral responsibility for distributed moral actions. Philosophical Transactions Of The Royal Society A: Mathematical, Physical, And Engineering Sciences, 374(2083), 20160112. doi: 10.1098/rsta.2016.0112
