Better Internal Audits with Artificial Intelligence
Auditing internal reports is a very manual process. We saw an opportunity to use artificial intelligence to improve that process. This article is all about how we did that, and where we are going with the technology.
Let’s start with describing what an internal audit really is:
- Internal audits are mandatory
- Humans review a large corpus of unstructured data looking for risk and controls
This sounds like a good problem to address with artificial intelligence. But wait… Is this complex task even possible to automate?
Yes it is.
The rest of this article is about how we went from knowing nothing about audits to building an audit enhancement tool, starting from requirements gathering from subject matter experts, progressing to problem statements, and finally landing at a solution that helps auditors to do their thing.
My first experience with internal audit was in the medical devices space (nuraleve.com), where we undergo regular ISO, Health Canada, and CE audits. Internal audit is a standard part of the medical devices manufacturing and distribution process on a global basis, and so Mathieu Lemay and I already had some exposure to the structure and function of an internal audits. What we didn’t see while working on the medical devices internal audits for a small company was the scale of internal audit at the enterprise level. It’s absolutely enormous.
The internal audit function of many organizations — especially at public companies and governments — is gigantic, mandatory, and not very effective. It took some deep work with audit professionals for us to fully understand the scope of the problem, and to implement the solution.
First Things First: Understanding Internal Audit in 3 Easy Steps
Like all good projects, we started with the help of our industry partners to briefly create a definition of Internal Audit. This led us to the use cases for applying deep learning.
Step 1: Why Internal Audits Exist
Gaining a fundamental understanding of the purpose of Internal Audit (IA) was a prerequisite before we could begin throwing deep learning around (AI for IA. hahaha) for this highly unstructured data. And so, we turned first to the Institute of Internal Auditors (IIA) for clarifications:
“The internal audit activity should monitor and evaluate the effectiveness of the organization’s risk management system.” — IIA-Standard 2110.A1
The documentation goes on to further explain:
“The internal audit activity should assist the organisation by identifying and evaluating significant exposures to risk and contributing to the improvement of risk management and control systems.” — IIA-Standard 2110 –Risk Management
So basically, IA is there to find problems. Now that we had this clear definition of IA from IIA, we set off to understand at both management and process levels, how audits are conducted.
Step 2: How Internal Audits Are Organized
How enterprises of all sizes achieve “evaluating the effectiveness of [their] risk management system” is through the performance of several internal assessments (“audits”) to get the pulse of various segments of their organization’s operations. We dug deeper into this, to outline these assessments’ underlying methods and processes, and came across this key structure of any risk management system: The Audit Universe. Each company defines the components of their Audit Universe by distinctly identifying:
- Corporate Structure, and accountability frameworks
- Internal subdivisions & External Components
- Programs of operations
- Activities within each program
Essentially, the boundary outlined by the Audit Universe identifies all things that can, and are subject to the supervision of the organization. It is within this audit universe that all internal audits and assessments are conducted, thus leading to step 3.
Step 3: Conducting an Internal Audit 101
Internal auditing, simplified, boils down to being an evaluation of how a segment of the organization performs when compared to their appropriate compliance or policy requirements. Similar to ISO, you say what you do, do what you say, and then verify that you did what you said you should do. Since internal audits are mandatory for all organizations’/groups’ intents and purposes, these activities are expected to adhere to a set of standards. Planning and undertaking these tasks requires a human’s review of a large corpus of unstructured data. This is where AI excels.
Auditors want to identify and understand the risks and controls in the operations they’re called to audit through a few different lenses. These variants in perspective are used to comb for potential issues in processes, and involves inspecting entities:
- By Control Objectives, Strategic objectives
- By Frequency of risk occurrence
- By Type
The reason for these different views is simply to look for problems. Notably, should a risk area be recurrently identified as consistently lacking for multiple years for which no control is being enforced, then this would be highlighted as an additional note to include in the management audit report. Similarly, if no risks or controls are identified, this can indicate to the auditor that there is a hidden risk within a program or process that is not being evaluated.
Oh No! Internal Audit is Broken
This manual process of doing internal audits is super broken. I described above what intrernal audits are supposed to do. As we worked to understand the nature of internal audit, we discovered an increasing amount of more and more breakages along processes, ranging from bias in audit selection and completeness to inadequate personnel training methods and highly manual labor intensive audit processes.
Here are a few examples we found in our requirements collection:
- Quantifiable overview is not available. Numbers are hard to generate, and so the 50,000 foot view is often misleading. The content’s corpus is generally sampled, as there is too much documentation to analyze manually.
- The process is dependant on non-quantifiable human decisions: Risks aren’t identified as they should, Controls are ineffectively associated. Not to mention the “Humans have feelings” factor; bias in both background and experience.
- Knowledge transfer is poor — leading to silos of knowledge and heavy reliance on key staff members.
- Some programs are over-evaluated while others are rarely audited. Example: credit card audits are too common, and management audits are too rare. If the only risks addressed in audit are financial risks, then other critical factors such as technology risk.
- The goal of the audit function is to demonstrate process performance and expose problems, rather than to just check a box and say you did an audit.
- The list goes on…
The main pain points for internal audit are audit completeness, better report investigation, and ensuring best practices during the audit. This grindingly manual process of doing internal audits required some significant rework, and was clearly an area that could benefit from our implementation of deep learning models.
How AuditMap.ai Improves Internal Audit
Having collected end-user requirements and identified flaws in the IA process with subject matter experts, our team was left with an opportunity to improve things. In short, we established our vision as, applying deep learning to internal audit, to bring better business outcomes without compromising audit quality.
AuditMap.ai is a natural language understanding tool that enables auditors to direct their assessments to known industry-wide issues: audit completeness, better report investigation, and ensuring best practices during the audit. Applying deep learning accelerates audit functions by resolving routine legwork bogging down the auditors’ ability to assess and report with no compromise on quality.
Consider a motivating example…
Example — As a Director, knowing what you’re working with
The ability to paint a high level picture for the audit function shouldn’t be hard. Right now, it is. It is the top-of-mind item of audit directors, new or established. AuditMap.ai uses a digital and interactive Audit Universe, and so knowing the lay of the land becomes the new standard. Say, for example, Acting Audit Director “X” needs a big-picture understanding of the risk environment associated to each of the programs under her supervision. Our tools provides the following key capabilities:
- Automatically identify risks, controls, and other key entities within all audit universe documentation.
- Map out a knowledge graph of entity relationships to accompany the risk summary.
- Map and relate risks and controls to documents within the audit universe
- Segment off the risk statements having high similarity, in an effort to identify areas of possible audit efficiencies, and
- Reveal at a roll-up of the risks by program, revealing where there are no audit documents or risk statements within the audit universe, indicating a program that has been ignored by the audit function.
AuditMap provides quantifiable overviews and puts the director and her team of auditors in a position where the critical focus of their efforts is placed on understanding the universe in which their work evolves, and working to deliver the work products the C-suite expects of them.
Having a profound understanding of an assessment scope can lead to more discoveries of efficiencies as more people involved in planning are empowered to ask questions along the lines of “How can our work be improved? Can it be simplified? Are there means being conducted outside of the scope of this program that we could leverage?”
Our solution required a lot of data labelling on sample audit reports, in order to form a basis for training a model to understand various types of sentences and their meaning. Applied deep learning lends itself quite well to benefiting the performance of a planning phase exploration of programs. Leveraging the capabilities suggested in the examples above, AuditMap.ai can use deep learning models to combine the results of risk and control reports created by AI-powered inferences indexed in an enhanced search engine (ElasticSearch with Kibana) capable of identifying text fragments of interest using syntax and/or semantics. Needless to say, this solution presents a leap forward in business processes automation for audit managers, leaving them more time to coach their teams and share their knowledge without consistently relying on key staff for mundane document understanding tasks.
Conclusion and Future Work
In conclusion, AuditMap.ai uses deep learning to accelerate a wealth of processes within the internal audit function. This holds especially true for contexts requiring a largely resource-intensive effort using digitized documentation, ranging from advanced search methods to document segmentation. In the case of internal audit, the way forward to better, faster, business outcomes is paved by applied deep learning.
The few examples demonstrated in the article above, among others, are fully operational and deployed in our AI-powered internal audit solution, AuditMap.ai
We are in the sales cycle now (finally) and our target market is audit shops, the big 5, and large enterprise clients as early adopter customers. To learn more about AuditMap.ai, just contact me and I’ll hook you up. Or contact Mathieu Lemay:
Discover your AuditMap, contact:
Mathieu Lemay
matt@auditmap.ai
T: +1–819–923–6288
If you liked this article on improving internal audit using A.I., press the follow button, and have a look at some of my most read past articles, like “How to Hire an AI Consultant.” In addition to business-related articles, I also have prepared articles on other issues faced by companies looking to adopt deep machine learning, like “Machine learning without cloud or APIs.”
-Daniel
daniel@lemay.ai ← Say hi. Ask about AuditMap.ai
Lemay.ai
1(855)LEMAY-AI
