What’s In the Box? AI Will Need to Explain its Decisions Before We Can Trust It.
XAI could allow a proliferation of AI technologies in sectors that require greater accountability, but we need to get to see inside the Black Box.
Early last December I found myself back in the job market searching for some temporary seasonal work during the Christmas break from university. My requirements for the role were not demanding: start time preferably post 8 a.m. (I had recently finished a 5-year stint of shift-work that I was not eager to renew); remuneration not necessarily exceeding that minimally stipulated by law and job site located within the geographical limits of the city in which I lived. Other than that I was game for anything. I responded to an online posting for an entry-level retail assistant position in a prominent high street fashion retailer. Without any unreasonable or immodest expectations about getting called for an interview, I began the application process in good faith.
Before any cover letter or résumé could be submitted I was required to complete a short questionnaire. This largely consisted of a vague, nebulous collection of 50 context-less nonsense questions apparently designed to assess a candidates ‘attitude’ and suitability for the role. I was asked to honestly use a 5-point rating scale to agree or disagree with a series of statements, such as:
“Some people make me feel nervous”,
“I get more good luck than most people”.
“I tend to assume the worse [sic]might happen”.
I don’t know how these scores were tabulated but I’m confident that any candidate who responds as instructed will not demonstrate an accurate portrayal of their ‘attitude’. When it is clearly stated that the applicant should not hedge their bets by selecting the ‘neither agree nor disagree’ option, how can any sensible person respond to an absolutist question like “when things go wrong I always look on the bright side”? The only logical answer to this question is ‘Strongly Disagree’ because not even the most optimistic person in the universe could ‘always’ look on the bright side. Even Christ himself despaired during his exile in the desert. However, ‘Strongly Disagree’ cannot sensibly be the correct answer in this case because there is no value whatsoever in asking a series of 50 semantically unclear trick questions to assess somebody’s suitability for a part-time job where the key responsibility is folding sweatshirts. This question is apparently meant to gauge a candidate’s resilience in the face of difficult situations, despite its nonsensical phrasing. So should an applicant submit an answer they know to be incorrect?
I didn’t get the job. I didn’t even get the chance to submit my CV, as the automated form tabulator instantly decided that I was temperamentally unsuited for the role of folding sweatshirts. I railed against the unfairness of the selection criteria. I was furious that I was not given any explanation about my answers and why they were incorrect. The injustice rankled.
In my case, this was a minor setback. I found another (better) job shortly afterwards. I earned enough pocket money to make sure there were presents under the tree on Christmas morning, with plenty left over for festive pints of Guinness. Christmas was saved. But for others, there may not be such a tidy resolution. More and more serious decisions regarding employment, loan applications, suitability for housing and besides are being made by automated systems that cannot offer explanations or justification. In the context of Artificial Intelligence applications, this is called the ‘Black Box’ problem and raises serious ethical and legal concerns.
The Black Box refers to the mysterious, secretive and unknowable processes that take place within an AI algorithm. Most AI systems follow the general path of an input layer (which might be a series of data points or images) that the program analyses, followed the middle layer, which consists of the algorithm itself (this is the part that does the analysing). Finally, we have the output layer — the decision that the program makes. Sophisticated mechanisms within the middle or ‘hidden’ layer of modern deep learning applications are by their very nature opaque and inscrutable. They may consist of many sub-layers within the hidden layer, passing information back and forth between them until the program figures out how to treat the data. Because the algorithm ‘learns’ with every new iteration or data point there is no way for an outside observer to follow what is happening within the hidden layer.
You could teach a machine-learning algorithm to recognize a picture of a cat, for example, by training it on different images of cats until it had enough data to correctly classify new images. However, it would not be able to tell you what in particular about the new image was cat-like. It may the shape of the ears, or the fur, or the whiskers that identify the cat to the algorithm, or a combination of many things. Or it may be some intangible example of ‘cat-ness’, imperceptible to human eyes, that the algorithm has picked up on. If this is the case then we will never know. A human would be able to explain without any difficulty why they have identified the cat, but the machine cannot.
As the uses for AI applications become more widespread it is becoming more important that we understand why systems make the decisions they make in certain circumstances. If an algorithm misidentifies a cat as a fish without any explanation it might result in an unhappy cat being housed in a fish tank if an animal shelter ever decided to automate its housing procedures. As automated systems find their way into healthcare applications there is a risk of misdiagnosis by machine, possibly caused by incomplete training data or other malfunction. If the incorrect AI decisions cannot be explained, the human designer of the program would not be able to understand why the incorrect decision was made or prevent it from happening again. In these situations, it is necessary for significant human oversight of the AI decision-making process.
One might think that there should be sensible human oversight of any AI application that takes decisions that have direct human consequences. In fact, there are plenty of automated processes operating today where decisions are being taken by solely by unsupervised algorithms that have serious real-world effects. Automated programs are used to calculate the likelihood of criminals re-offending which could affect their sentencing or parole conditions. Universities are using predictive algorithms to decide whether or not to offer prospective students places on courses. When the reasons behind these automated decisions cannot be adequately explained it makes it very difficult for somebody to appeal a decision they believe is unfair.
If the input data used by the algorithm is incomplete or the result of inferior or unreliable collection methods (as in the case of my poorly-written job questionnaire) and this leads to a disputed outcome it is imperative that there is an appropriate avenue of appeal for the subject of the decision.
Concerns like this are what has led to the publication of Article 22 of the EU’s complex G.D.P.R. legislation which deals with automated decision-making:
“The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her”.
This means that institutions like banks, governments or any organisation that utilises predictive modelling of human behaviour to inform decision-making cannot rely solely on the automated process in cases where a person is the subject of the decision. In other words, every person has the right to explanation. If the automated processes are opaque and unexplainable, as most conventional AI processes are, it becomes extremely difficult for institutions to rely on powerful and useful tools in the case of disputes.
When a human decides to make a financial investment, for example, or terminate an employment contract, they are legally and morally accountable for that decision. An automated AI system operates independently of the human designers who constructed it; once it’s running it makes decisions on its own, with no accountability or liability whatsoever. This is a hugely problematic concept when we consider wider issues of justice or democracy.
These kinds of considerations have contributed to the recent development of a sub-category of AI called Explainable AI (or XAI) which is concerned with developing techniques that allow machine learning processes to become more transparent. The concept is still in its relative infancy but given time it could allow wider adoption of AI processes in areas where accountability and explainability are serious concerns.
In my next article, I will take a look some XAI techniques that may begin to come on stream over the next few years, some of the sectors where it may be particularly useful, as well as some of the movers and shakers driving progress in the field. I’ll also talk about the limitations of the technology and the challenges it faces.
All views are my own and not shared by Oracle. Please feel free to connect with me on LinkedIn
