"Any sufficiently advanced technology is indistinguishable from magic" – Arthur C. Clarke
With the advances in self-driving cars, computer vision, and more recently, large language models, science can sometimes feel like magic! Models are becoming more and more complex every day, and it can be tempting to wave your hands in the air and mumble something about backpropagation and neural networks when trying to explain complex models to a new audience. However, it is necessary to describe an AI model, its expected impact, and potential biases, and that’s where Explainable AI comes in.
With the explosion of AI methods over the past decade, users have come to accept the answers they are given without question. The whole algorithm process is often described as a black box, and it is not always straightforward or even possible to understand how the model arrived at a specific result, even for the researchers who developed it. To build trust and confidence in its users, companies must characterize the fairness, transparency, and underlying decision-making processes of the different systems they employ. This approach not only leads to a responsible approach towards AI systems, but also increases technology adoption (https://www.mckinsey.com/capabilities/quantumblack/our-insights/global-survey-the-state-of-ai-in-2020).
One of the hardest parts of explainability in AI is clearly defining the boundaries of what is being explained. An executive and an AI researcher will not require and accept the same amount of information. Finding the right level of information between straightforward explanations and all the different paths that were possible requires a lot of training and feedback. Contrary to common belief, removing the maths and complexity of an explanation does not render it meaningless. It is true that there is a risk of under-simplifying and misleading the person into thinking they have a deep understanding of the model and of what they can do with it. However, the use of the right techniques can give clear explanations at the right level that would lead the person to ask questions to someone else, such as a data scientist, to further their knowledge. The key to this process is to have effective conversations and communications to make sure the necessary information is conveyed.
How can one gain experience in effective communication? Contrary to common belief, gaining practice in explaining does not require reaching a senior position. While it is true that skills such as explaining complex concepts improve over time through trial and error, junior employees are often extremely effective at it as they have just learned the subject. The key is to get experience in explaining to non-technical audiences through practice and anyone can do that without needing to wait to become senior. In fact, understanding a complex concept and being able to explain it to a non-technical audience are not mutually exclusive. To improve this skill, there is only one recipe: practice, practice, and practice.
Explaining complex concepts can be challenging due to something called the curse of knowledge. It requires patience and repetition from different angles in order to create a lasting memory. Generative AI is becoming increasingly accessible to the public through large language models, and this has created a need for understanding. There are concerns about ChatGPT giving wrong information, but it is important to understand why in order to comprehend the capabilities and limitations of the technology. We are all familiar with predictive texts on our phones and emails, and large language models are doing the same process but on a larger scale. Like our phones a decade ago, they don’t always predict the next word correctly. Looking back at the numerous advancements in technology, it is clear that everything has been incremental, and using this incrementality is key to explaining concepts that would otherwise seem like magic.
Explainability in AI is not only important when explaining a concept to others; it can also be challenging to add explainability to existing machine learning models. When deciding on a model, the needs and end user should be taken into account to ensure a trade-off between complex models and explainability is considered. Sometimes, the simplicity of a linear regression outweighs the complexity of a more robust model. Decisions that have a material impact on someone’s life, such as authorizing a bank loan, require an explanation. In particular, information is invaluable when the output of a model is not the desired one. Having an explainability process in such cases can uncover flaws in the model or even in the training data used.
To conclude, explainability in AI occurs at different stages. Explaining the concept to the end-user to ensure they understand the potential limitations. Explaining a model to peers and non-technical audiences to understand the ins and outs of the algorithms. Explaining the decisions resulting from the model applications to ensure it follows regulations and that no implicit bias is present. All of these three areas are essential to the development of AI and if one of these aspects seems too complicated for the problem you are trying to solve, it may be worth considering if the model being used is the best one.
If you enjoyed this story, don’t hesitate to clap or to reach out in the comments! Follow me on Medium for more content about Data Science!