Notes from Industry
_by Alexander Petrov, John Thomas, Ricardo Balduino, Maxime Allard, and Aakanksha Joshi_

Introduction
Machine Learning (ML) applications have become ubiquitous. News about AI for self-driving cars, online customer support, virtual personal assistants, and so on come daily. And yet, it may not be obvious how to connect existing business practices with all these amazing innovations. A frequently overlooked area is the application of natural language processing (NLP) and deep learning to help process huge volumes of business documentation quickly and effectively to find the proverbial needle in the haystack.
One of the domains that allows organic application of ML is risk management for financial institutions and insurance companies. There are many questions that organizations face regarding how to apply ML to improve risk management. Here are just a few of them:
· How to identify impactful use cases that can benefit from using artificial intelligence?
· How to bridge the gap between intuitive expectations of subject matter experts and capabilities of technology?
· How to integrate ML into an existing enterprise information system?
· How to control the behavior of ML models in a production environment?
This article aims to share the experiences of the IBM Data Science and AI Elite (DSE) and IBM Expert Labs teams, based on multiple client engagements in the risk controls area. IBM DSE has built various accelerators which can help organizations jump start their adoption of ML. Here, we will go through use cases in the risk management space, introduce a cognitive risk controls accelerator, and discuss how machine learning can transform enterprise business practices in this space.
Risk Management Sketch
In 2020, multiple financial institutions were hit with fines exceeding hundreds of millions of dollars per individual organization. The reason for the fines was an inadequate risk controls state.
This triggered a call for financial companies to ensure high quality of the large numbers of risk controls they have to work with. This includes explicitly identifying risks, implementing risk controls to prevent risk development, and finally establishing testing procedures.
For non-specialists, risk control is a bit perplexing. What is this about? A simple definition is that risk controls are put in place to monitor risks for a company’s business operations. E.g., a security risk may be that an intruder guesses a password and as a result gets access to someone’s account. A possible risk control may be designed as establishing a policy that requires long and non-trivial passwords enforced through the organization’s systems. As a consequence of the Sarbanes-Oxley Act (SOX), public companies require means to efficiently manage such risks and, as part of this effort to build risk controls and assess the quality of these controls.
An important element for risk managers is whether the controls are well defined. The assessment for this may be done through answering questions like who monitors the risk, what should be done for risk identification or prevention, how often the control procedure should be done in the organization’s life cycle, etc. All these questions should be answered. Now we need to realize that the number of such controls in an enterprise is from thousands to hundreds of thousands and it is very difficult to make an assessment of the controls corpus manually. This is where contemporary AI technology is able to help.
Of course, this type of challenge is just an illustration and it would be impractical to attempt covering the vast area of Risk Management within a single article, so we focus on a few specific challenges that practitioners face in their day-to-day practice and that were already implemented using the Cognitive Risk Controls accelerator.
There are not many public risk controls databases available, so the solution in the accelerator is based on NIST Special Publication 800–53 for security controls that is available at https://nvd.nist.gov/800-53. This security controls data base is small, but it allows us to demonstrate the approaches that can be scaled to large volumes and different domains of risk controls.
Using Text Analytics and Deep Learning for Risk Controls
One of the key use case categories is to rationalize the existing risk controls: the challenge is that there may be a lot of historic aspects to how the existing risk controls were developed. E.g., some risk controls may be built by copying other existing controls with minimal modifications. Another example is that some risk controls may be formed by integrating multiple risk controls into one. Common consequences of this approach are duplicated controls and the presence of controls that are not relevant to the business any longer. One of the most difficult challenges is to assess the general state of quality for the existing risk controls. Hence, the first target from a business perspective is to build quality assessment: automatically assessing the quality of control descriptions saves huge time on routine reading of descriptions by focusing only on those which are really important to review and improve. A good question is how AI comes into the picture here. NLP-based ML models have become very effective at common language-related tasks and, in particular, at challenges such as answering questions. One type of model that can be referenced here is based on the Transformer architecture (for more details, please see an article about Transformer architecture at https://medium.com/inside-machine-learning/what-is-a-transformer-d07dd1fbec04).
In the risk management sketch, the ability to answer questions about a risk control description was key to assessing the quality of the control’s descriptions. From a birds-eye view, the number of the unanswered questions is a good indicator of the quality of a control description. The best news is that with the capabilities of contemporary AI models such as Transformers and with additional practical rules, this technique of asking the right questions becomes an effective mechanism to control a large volume of control descriptions by a small team with the assistance of AI.

Frequently, finding duplicates in documents is considered a straightforward task and Levenshtein Distance can help find items expressed with similar wording. However, this becomes a much more challenging task if we would like to find semantically similar descriptions. This is another area where contemporary AI can be helpful – the embeddings built using large neural networks (e.g. autoencoders, language models, etc.) can capture semantic similarity. From a practical outcomes perspective, our experience has been that the identification of duplicates and overlaps may lead to reducing the volume of controls up to 30 per cent.

Additionally, it became a common practice to analyze the internal structure of information through ML techniques such as clustering. This allows the business practitioners to better understand the content of controls on a larger scale and to see whether the existing taxonomy for risks and controls is well aligned with the content, or what may be missing in both.

The previous use cases were focused on the analysis of existing controls. Another use case focuses on helping risk managers create new risk controls. Recommending controls for a given risk using semantic similarity can significantly reduce manual effort and provide a flexible template for building controls. Machine learning can help here with analysis of the risk description and figuring out the right set of controls to address each risk.
In large organizations, it is typical that teams work on solutions and best practices which may be used by other teams. Adopting best practices across the organization requires extensive training. Machine learning can be very useful in such situations. An example may be classification of controls as preventive or detective. In this use case, we use supervised machine learning to extend the classification of controls to the whole set of controls by using the existing labeled set from a particular team, i.e. knowledge transfer is done using machine learning as opposed to time consuming training of personnel.
Cognitive technology in the IBM DSE risk controls accelerator allows us to structure the risk controls, to recommend controls for the risk formulated in natural language, to identify overlaps within the controls, and to analyze the quality of controls.
The accelerator delivers a cognitive controls analytics application that integrates the developed models and applies them to unstructured risk controls content.
Cognitive Risk Controls Implementation Using IBM Cloud Pak for Data
Logically, the Cognitive Risk Controls Accelerator contains several components:
- The first one is a so-called cognitive assistant – it is an application that applies ML models to facilitate content processing, e.g., by identifying the risk control priority, category, and assessing the quality of the control description. As part of productization, a cognitive assistant becomes a part of the enterprise informational system.

- The second component is content analysis: when the data is enriched via Machine Learning models, Watson Discovery content mining can be used to find insights in the enriched content

- Yet another component is a set of Jupyter notebooks that support Data Science models

Let’s look under the hood of the accelerator-based implementation using IBM Cloud Pak for Data.
Before we do this let’s briefly review the IBM platform and approach. IBM has a prescriptive approach to the journey to AI called the AI ladder. In his "AI Ladder: Demystifying AI Challenges" Rob Thomas (SVP, IBM Cloud & Cognitive Software) substantiated that to turn your data into insights your organization should follow the phases listed below:
• Collect – ability for easy data access, including virtualizing the data
• Organize – the means to cataloguing your data, building data dictionaries, and ensuring rules and policies on accessing data
• Analyze – this includes delivering the ML models, using data science for identifying the insights using cognitive tools and AI techniques. This naturally requires building, deploying, and managing your machine learning models
• Infuse – from a lot of perspectives, a key phase. This refers to the ability to operationalize AI models in a way that allows the business to trust the outcomes, that is, to use your machine learning models in enterprise systems in a production mode while being able to ensure ongoing performance of these models and their explainability.
Cloud Pak for Data is IBM’s multi-cloud Data & AI platform delivering an information architecture and providing all the outlined capabilities. The following diagram captures the details of developing an implementation in the context of the AI Ladder.

It captures the phases of implementing a cognitive risk controls project based on the DSE accelerator:
- The first two phases in implementing a risk controls project are acquiring and cataloguing the data set – as an example, in the accelerator we are using the NIST controls data set. Controls here are expressed as free text descriptions.
- The next phase is the enrichment of the acquired unstructured data which is done in Watson Studio: clustering is used as a way to understand the internal structure of content. The risk control narrative may be quite long and multiple topics may be discussed, so some mechanism may be required to track changing topics as the description progresses. In our practice for clustering, we used both K-means on top of embeddings and Latent Dirichlet Allocation (LDA). It does require careful coordination of data scientists and subject matter experts as the mathematics may not align ideally with the expectations of SMEs. A wider range of enrichments is also possible here – a good example is classifying the quality of descriptions.

- When the enrichment is finished, we need to understand the resulting data set. This leads us to the Exploration phase. In practice, the challenge is volume; content review is one of the most time-consuming processes as it requires careful reading of a large volume of text. How can we explore huge volumes of unstructured information? Watson Discovery content mining is the tool that makes this possible and greatly reduces the effort.
- After the content is reviewed by SMEs, it forms the basis for building supervised machine learning models. The IBM platform provides the means to deploy the models, monitor drift, and get explainability for the decisions made by complex models. All of this is covered by the operationalization of machine learning and supported by IBM Cloud Pak For Data.
Conclusion
This article introduced one of the growing areas of application of machine learning in contemporary business – cognitive risk controls. Visit our Accelerator Catalog to learn more about the Cognitive Controls Accelerator. Please do not hesitate to reach out to the IBM Data Science and AI Elite Team if you are interested in knowing more about cognitive risk controls and AI technology. Also, please contact IBM if you see that your use cases are similar to the ones presented, or if your business and technical challenges may be addressed by the mentioned approaches or tools.
Acknowledgements
Authors (IBM DSE and Expert Labs) express gratitude to their colleagues for the continued collaboration and development of the business and technological approaches for cognitive controls: Stephen Mills(Managing Director, IBM Promontory), Miles Ravitz (Sr. Principal, IBM Promontory), Rodney Rideout (Delivery Executive, IBM Global Business Services), Vinay Rao Dandin (Data Scientist), Aishwarya Srinivasan (Data Scientist, IBM DSE), and Rakshith Dasenahalli Lingaraju (Data Scientist, IBM DSE).