How to evaluate expected benefit of implementing a model
Often, in data organisations, stakeholders have to make decisions about ML strategy. Choices like whether to pursue the development of a new model, decide on implementing a new model incurring in , hopefully small, deployment costs or buy information from data providers are frequently taken based on intuition alone.
In this article, based on vanilla assumptions about scenario, economical model the company operates in and shape of ROC curves, I try to shed some light on this topic. The goal is to have a function calculating the marginal expected profit gain of replacing a strategy based on a model with a given AUC to a model with different AUC.
Economic Model
In this first iteration of the study, we approximate the business model as a game where the company earns a fee when the customer does not default and loses a unitary price otherwise.
The company can chose whether or not to sell the product to any customer. Let’s define condition positive as the event where the customer defaults and predicted positive as the event where the company denies the operation.
The expected profit is:
Expected profit = Probability true negative × fee -Probability false negative -Probability false positive × fee
One false positive costs fee as the opportunity cost and one false negative costs 1 because the price is unitary.
From AUC to ROC Curve
The intention is to build a ROC Curve given an AUC. We assume all ROC Curves is a regularized beta function with parameter a set to 1 and the parameter b is such that the area of the curve is equal to the AUC; this means b = AUC/(1-AUC) .
Below we show some examples of these curves:

From Operational Point to profit
In a scenario with constant population bad rate one can, based on an operational point – (False Positive Rate, True Positive Rate)a point in ROC space-, get all the probabilities needed by the economic model to calculate the agent expect profit:
Probability true negative = False Positive Rate × bad rate Probability false negative = (1 - True Positive Rate) ×bad rate Probability false positive = False Positive Rate ×(1 - bad rate)
If the company has a model with given AUC it can chose the optimal operational point as the one that maximizes expected profit.
Expected profit increase
With the pieces laid so far we can build a function from AUC to expected profit.
A company, in a given scenario -bad rate and fee-, facing questions on whether to change its model based strategy for a new algorithm with promised better AUC, can use this function as an input to the decision. The output of this function based on the AUC of current model minus the output based on the new model AUC is the expected profit increase.
Let’s study how different factors contribute to expected profit increase by investigating some example scenarios.
As a first example, let us see the relative profit increase from using no model – AUC=0.5 – , in a scenario with fee 0.2 and different bad rates.

Some remarks can be made:
- Significative increase in profit does not happen in every situation for any AUC gains.
- In scenarios of low bad rates the AUC gain should be very significant to justify a change in model.
Being more concrete: in a scenario with fee 0.2 and bad rate 10% it is only advantageous to switch from a strategy of using no model and not filtering out any operation when a new model has at least AUC of 0.79.
The maximum price for a query on a service providing a model of AUC 0.85 in this situation is 0.009 times the price of the product.
Next, we plot expected profit increase when changing models for different scenarios:

Some further conclusions:
- The tendency of only being advantageous to switch models when some critical performance is reached keeps on.
- In the analysed space the results seem to be more sensitive to fee than to base AUC.
Critical AUC
On the examples we saw that a certain AUC threshold should be reached to only than switching models being a sensible decision. Let’s call this critical AUC: the minimum value in a given scenario that if outperformed by a model then profit increase can be seen.
Below we see how the critical AUC varies across bad rate for different fees in a scenario of base AUC=0.5.

Observations on this result:
- For a fixed fee, only in one specific bad rate scenario, any Model Performance gain increases profit. In this scenario the company would be breaking even.
- For low bad rates the new model performance should be very high to justify changing the naive strategy of simply not filtering out any operation.
- The lesser the fee the easier it is to achieve critical AUC.
- For bad rates close to 1 the model also should be very good to replace the naive strategy of denying any operation.
Caveats
- The AUC considered in this study is calculated from the whole population. This, in practice, is very hard to get: we can only observe the outcome of an event for people who actually took the product. This is why no considerations about new people becoming eligible for operations are made.
- We also consider the fee will remain the same when changing strategy. The possibility of being able to reduce fee to increase volume is, then, omitted.
- The family of ROC Curves considered is small. Although I believe the shapes of the curves will not impact significantly the overall results it has yet to be confirmed with further analysis.
- We consider the models to be perfectly continuous. Concentration on some predictions are common in model development and might change the results.
End Notes
This study has no pretension whatsoever of being complete. If you see something that could be addressed better or any ludicrous assumption please ping by any means or, better yet, fork the repo I’m using for this study and make adjustments.
If you want to explore by yourself the results, in the repo the function to calculate the expected profit from switching from a model with base AUC to a different one is implemented.