Individual recourse for Black Box Models

Model Interpretability

Intuitively explained through a tale of cats and dogs

"You cannot appeal to [algorithms]. They do not listen. Nor do they bend."

– Cathy O’Neil

In her popular book Weapons of Math Destruction Cathy O’Neil presents the example of public school teacher Sarah Wysocki, who lost her job after a teacher evaluation algorithm had rendered her redundant O’Neil (2016). Sarah was highly popular among her peers, supervisors and students.

This post looks at a novel algorithmic solution to the problem that individuals like Sarah, who are faced with an undesirable outcome, should be provided with means to revise that outcome. The literature commonly refers to this as individual recourse. One of the first approaches towards individual recourse was proposed by Ustun, Spangher, and Liu (2019). In a recent follow-up paper, Joshi et al. (2019) propose a methodology they termed REVISE, which extends the earlier approach in at least three key ways:

1. REVISE provides a framework that avoids suggesting an unrealistic set of changes by imposing a threshold likelihood on the revised attributes.
2. It is applicable to a broader class of models including Black Box classifiers and structural causal models.
3. It can be used to detect poorly defined proxies and biases.

For a detailed discussion of these points you may check out this slide deck or consult the paper directly (freely available on DeepAI). Here, we will abstract from some of these complications and instead look at an application of a slightly simplified version of REVISE. This should help us to first build a good intuition. Readers interested in the technicalities and code may find all of this in the annex below.

From 🐱 to 🐶

We will explain REVISE through a short tale of cats and dogs. The protagonist of this tale is Kitty 🐱 , a young cat that identifies as a dog. Unfortunately, Kitty is not very tall and her tail, though short for a cat, is longer than that of the average dog (Figure 1).

Much to her dismay, Kitty has been recognised as a cat by a linear classifier g(x) that we trained through stochastic gradient descent (once again interested readers may find technical details and code in the annex below). Figure 2 shows the resulting linear separation in the attribute space with the decision boundary in solid black and Kitty’s location indicated by a red circle. Can we provide individual recourse to Kitty?

Let’s see if and how we can apply REVISE to Kitty’s problem. The following summary should give you some flavour of how the algorithm works:

1. Initialise x, that is the attributes that will be revised recursively. Kitty’s original attributes seem like a reasonable place to start.
2. Through gradient descent recursively revise x until g(x*)=🐶 . At this point the descent terminates since for these revised attributes the classifier labels Kitty as a dog.
3. Return x*-x, that is the individual recourse for Kitty.

Figure 3 illustrates what happens when this approach is applied to Kitty’s problem. The different panels show the results for different values of a regularisation parameter that governs the trade-off between achieving the desired label switch and keeping the distance between the original and revised attributes small. In all but one case, REVISE converges: a decrease in tail length along with an increase in height eventually allows Kitty to cross the decision boundary. In other words, we have successfully turned Kitty into a dog – at least in the eyes of the linear classifier!

We also observe that as we increase the regularisation parameter for a fixed learning rate, REVISE takes longer to converge. This should come as no surprise, since we explicitly regularise more strictly with respect to the penalty we place on the distance that Kitty has to travel. When we penalise too much (bottom right panel), Kitty never reaches the decision boundary, because she is reluctant to change her characteristics beyond a certain point.

Discussion

While hopefully Kitty’s journey has provided you with some useful intuition, the story is of course very silly. Even if your cat ever seems to signal that she wants to be dog, helping her cross that decision boundary will be tricky. Some attributes are simply immutable or very difficult to change, which Joshi et al. (2019) do not fail to account for in their framework. Their proposed methodology offers a simple and ingenious approach towards providing individual recourse. Instead of concerning ourselves with Black Box interpretability, why not simply provide remedy in case things go wrong?

To some extent that idea has its merit. As this post has hopefully shown, REVISE is straight-forward to understand and readily applicable. It could be a very useful tool to provide individual recourse in many real-world applications. As the implementation of our simplified version of REVISE demonstrates, researchers should also find it relatively easy to develop the methodology further and tailor it to specific use cases. The simpler version here, for example, may be useful in settings where the dimensionality is relatively small and one can reasonably model the distribution of attributes without the need for generative models.

Still, you may be wondering: if the original classifier is based on poorly defined rules and proxies, then what good does REVISE really do? Going back to the example of high-school teacher Sarah Wysocki, one of the key attributes determining teachers’ evaluations was their students’ performance. Realising this, some teachers took the shortest route to success by artificially inflating their students’ test scores. That same course of action may well have been suggested by REVISE. As Joshi et al. (2019) demonstrate, this very property of REVISE may actually proof useful in detecting weaknesses of decision making systems before setting them loose (key contribution 3).

Nonetheless, the example above also demonstrates that approaches like REVISE, useful as they may be, tend to provide solutions for very particular problems. In reality data-driven decision making systems are often subject to many different problems and hence research on trustworthy AI will need to tackle the issue from various angles. A few places to start include the question of dealing with data that is inherently biased, improving ad-hoc and post-hoc model interpretability and continuing efforts around causality-inspired AI.

References

[1] Joshi, Shalmali, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh, Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems (2019), arXiv Preprint arXiv:1907.09615.

[2] O’Neil, Cathy, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (2016), Crown

[3] Ustun, Berk, Alexander Spangher, and Yang Liu, Actionable Recourse in Linear Classification (2019), Proceedings of the Conference on Fairness, Accountability, and Transparency, 10–19

Annex

In my blog posts I aim to implement interesting ideas from scratch even if that sometimes means that things need to undergo some sort of simplification. The benefit of this approach is that the experience is educationally rewarding – both for myself and hopefully also for readers. On this Medium publication I will attach only the commented code for the linear classifier and REVISE below. For code with syntax highlighting and to also get a flavour of the mathematics involved, checkout the post on my personal blog.

Linear classifier

Linear classification is implemented through stochastic gradient descent (SGD) with Hinge loss. The linear_classifier function outputs an S3 class, which is common way to use object-oriented programming in R. What follows are just print and predict methods for the classifier.

REVISE (simplified)

As flagged above, we are looking at a slightly simplified version of the algorithm presented in Joshi et al. (2019). In particular, the approach here does not incorporate the threshold on the likelihood nor does it account for immutable attributes.

The revise function is implemented as a method for the classifier class created above.

---

Originally published at https://pat-alt.github.io on April 26, 2021.