Data quality is a business matter

Opinion

Data quality may be a term coined and celebrated by us data management fanatics, but it is a business matter. At the end of the day, an organisation’s hard working employees gain the most from data quality improvement initiatives. I think that cannot be stressed enough. In this blog, I’d like to highlight the steps to in approaching data quality from a business perspective.

1. Identify your business bottlenecks

Take a look at the bottlenecks within your business processes. Listen to those hard working employees, hear them out. What are their frustrations? What are the issues they have to deal with? Break it down. For example when an employee has to wait because they haven’t received the correct price list. Or a business partner cannot be reached because his phone number can’t be found. Or a customer is complaining because they’ve been charged the wrong price. Although these sound small, they matter at scale. Executive management will understand why these are issues – they are business issues. However, all of them are rooted in low data quality. But that shouldn’t be the focus point – focus on the business.

2. Define your framework

The next step is to define your framework. What are different dimensions that matter to your business? Approach this from a business perspective. Use these dimensions to report on the progress of your improvement initiatives. Take the existing problems and break them down. Below are some problem statements and their relevant dimensions to consider.

"I received the wrong information…"

Accuracy: information should reflect reality. If your inventory says you’ve got 10 items left, there should actually be 10 items on the shelf.

"I am waiting to receive the right information…"

Timeliness: Data should be available when it’s needed. If someone has to wait for the right information to come available, you’re losing efficiency.

"I can’t contact them because I don’t have their phone number"

Completeness: if an email address and phone number are required to contact business relations, these should always be filled out when establishing new relationships.

"But my manager that pricing wasn’t right…"

Consistency: A field should contain what it’s supposed to contain. If someone is looking for a sales price, they should find the sales prices with the right margin.

"Our customers unsubscribe from our mailing list because we send it too often…"

Unicity: Information should be stored once. There is no point in maintaining the same information in multiple places, let alone in the same system. One place, means less work to maintain.

"The intern made unexpected last minute changes…"

Integrity: Just like in journalism, you should be able to trust the source of your information. So make sure that only the authorised people are allowed to edit it.

3. Have IT support you

So you’ve defined the challenges and the relevant dimensions that matter to your organisation. Have IT support you in enforcing those standards in your IT applications. Some key aspects to consider as highlighted below.

• Input controls: if you know what you expect to be entered in a specific field within an application, make sure that only that can be entered. Like a sales price, should be a number in the correct currency. If a user enters something that doesn’t match the set criteria, then it cannot be saved. If employees start complaining about these constraints, listen to them. That means that they have additional requirements: they probably don’t have a place to store specific information. An example I encounter a lot is with names in CRM systems. They are often ‘polluted’ with information about their status, like ‘inactive’ or ‘use number 10230 instead’ in the name field. It would help to archive or delete these old items.
• Reference tables: if multiple people use the same information, set up reference tables to support them. These are super easy to create and can bring so much value to the business. They can be as easy as a word document bookmarked in your browser. Think about a reference table for currency exchange rates, product codes or country codes. Looking for something, quickly browse to the bookmark to find the right information.
• Data Profiling on data stores: data profiling processes are becoming more and more standardised and available. We used to carry these out quickly using our own SQL scripts, but PowerBI has had a standard data profiling feature for a while now. Data profiling is the process of identifying what your data looks like: so what values are available, what characters do they contain, how often does the same value appear. For instance, does a date have the format 17/02/2021 or 2021.02.17?

4. Assess the extent to which information meets your business requirements

You started out by defining what the bottlenecks were. Keep coming back to those. To what extent have those bottlenecks been resolved? Do issues still arise? Are employees still waiting for information? Or having to make multiple phone calls to confirm information? Are they still losing time looking for information? If so – use data profiling reports to determine where the gaps are. If email addresses and phone number are required information, check how many of them are missing using a data profiling completeness check. Formalise this responsibility: have someone do this on a periodic basis. Make it part of their responsibilities. Assign problem owners and empower them to set up solutions for these problems. Have someone – on the ground, who is actually suffering from the problem – keep track of this. If your organisation has assigned process owners, they are a great place to start.

5. Cleanse data at the source

When having to fix issues quickly, we often tend to choose for ‘quick fixes’. After having made 5 phone calls to figure out what that business partner’s phone number was, the next pressing tasks comes and and the phone number isn’t stored in the CRM system. So next time, someone else will likely face the same issue. Don’t let that happen. Fight for permanent solutions. Don’t have multiple people have to go through the same hassle of finding correct information. Once that is done, make sure that it corrected at the source.

Information that your organisation collects and stores should be accurate. It should add value. Say something meaningful. Data quality often doesn’t become an issues until some type of data analysis or data science project comes up – because the data analyst or scientist starts using the data for something other than it was initially intended for. If that’s the point at which it becomes clear, it’s too late. The chances are high that the business is encountering these issues way earlier than that, but they aren’t given the means to address them.

I hope this blog gave you some clarity on why data quality is a business matter and how you can tackle it. Let me know what you think!