dq:view - Steve Tuck on Data Quality

dq:View has moved...

Hi folks,

Thanks to everyone that has been following dq:View, but I've now taken the decision to move the blog to Datanomic's corporate site. I hope you'll continue to follow my witterings at http://www.datanomic.com/category/resources/blog/ or via an appropriate feed; RSS or Atom.

You can also find me online on LinkedIn as http://www.linkedin.com/in/SteveTuck, on Plaxo as http://stevetuck.myplaxo.com and if you're into Twitter, you can follow my Tweets at http://twitter.com/SteveTuck.

All the best,

Steve

p.s. There are already 2 new entries for you to read on data quality related topics at http://www.datanomic.com/category/resources/blog/.

Getting their wires crossed

What on earth possessed me to transfer my mobile, office and broadband lines in the same week? How could I be so naive as to think that it would all go smoothly?

After a lot of frustrating calls to premium rate numbers, working through countless automated menus and listening to a lot of dreadful "on-hold" musak, my office line did get successfully transferred, but the mobile and the broadband were not so successful. Both look set for a delay of at least a week.<br>

What has frustrated me most in my dealings with the FOUR telephone companies involved is the failure of anyone to take any ownership or responsibility for resolving THEIR issues. Instead I have been passed from pillar to post in my quest to sort things out for them.<br>

It seems that everyone I speak to has a very limited remit and is incapable of talking to their colleague or even transferring me to the next department. Why do that when you can bump the customer and give them another premium rate number to call which places them in another queue they have to endure?<br>

I now know more about the internal workings of the ordering process at these companies than most of their staff do and certainly more than I care to. Each of them has some form of single customer view in place and references me by my telephone and account numbers, but none of them appear to be truly joined up and the customer service staff have access to information that is incomplete at best and frequently incorrect. They may have the same identifier on the records, but their systems and processes are certainly not working in a joined up way, leaving the customer to assemble their own single view of the enterprise when things go wrong. As far as I can see, the only saving grace for these organisations is that they are all as bad as the others. Perhaps they should be called miscommunication companies...

dn:Director - a fresh approach to data quality

Why do so many organisations turn a blind-eye to data quality?  One thing for sure is that the legacy data quality software providers have done little to help address this crucial business issue by delivering products that require years of expertise to successfully leverage all of the functionality available (and, just as importantly, to know when to use something else instead).  After a dozen years of working in the field, and having built a highly profitable consultancy business to help clients address this short-fall, I decided a year or so ago to join Datanomic.  I'm delighted to say that, last month, we celebrated the launch of dn:Director, a data quality product that is setting new standards for data quality management in the 21st Century.

I've been privileged to work on data quality projects with many leading, blue-chip companies over the years, but one of the things that struck me was that I was being asked the same questions by clients in 2004 as I was asking myself more than a decade earlier; they were identifying the same old deficiencies in data quality products and having to employ the same workarounds to resolve them.  Sure, the vendors have done something to smarten up the look of their software, but, under the covers sits essentially the same code that was initially developed for mail-room efficiency in the 1980's.

Two more things struck me:

1. All of the software vendors talked about delivering a tool for "business users" but the reality was that just about every project relied on the IT department to develop the business rules.
2. Because of the complexity of using the software to good effect, the cost and duration of projects was prohibitive; the reason I was working with so many blue-chip companies was that they were the only ones that could afford to undertake such major projects!

These were the things that motivated me to create Tranato and subsequently to join Datanomic in 2005 and bring together the two technologies under a shared approach.  Put simply, we feel that a data quality product needs to be much more accessible - you shouldn't need to be a software guru to get value from it.

dn:Director is the result of many years experience in data quality and data management; not just my own, but that of people like Gerry Kelley (Datanomic's VP of Professional Services) and his team, and the shared experiences of our clients and partners.  Taking Datanomic's approach (The Four Cornerstones) and methodology as its foundations, dn:Director has been built from the ground up, using the best-available modern technology.

Developing dn:Director in Java and using standards-based interfaces (such as JDBC, JMS and XML) has enabled us to deliver a technically advanced and extensible data quality product that supports both batch and real-time processes (providing data quality services through SOA).  But the thing that everybody notices first is just how easy it is to use - you should hear what out customers and partners have had to say about it:

"This is great - it's so easy understand and configure business rules"

"I love the way that you can build rules from the data - it's so quick and intuitive"

"This will halve the time it takes to deliver a project"

For more information visit Datanomic's website or call on +44 (0)1223 228400.

Note: I know this is very commercial for a blog entry, but given the amount of personal time, energy (and money) I've committed to making dn:Director a success, I hope you'll forgive me.

Talking Dirty

Many people (including acknowledged data quality gurus) appear to have a very restricted view of what constitutes "dirty" data and what you can do to improve it.  I was reading an article recently that expounded the case for cleaning-up dirty data, but never ventured beyond tried and tested examples of historical data and alternative versions of names.  In my experience, dirty data can contain a host of hidden knowledge - we need to think beyond the idea of merely cleaning data and understand how we can actually turn "dirty" data into an asset.

For example, take a look at most customer databases and you'll find evidence of how text fields, including names and addresses, are used to store additional pieces of information which are otherwise not catered for.  For instance, call centre staff will often use the customer name to store notes about when to call a customer, how to contact them or even personal comments about them:

• Mr Andrew Monkton - call after 8pm
• Ms Fiona Brookes (work - 1-555-2354-321)
• Mrs Angela Watson - husband is a miserable ****

Most data quality software provides little help in understanding and improving the quality of this data.  What's needed is the ability to profile and analyse the contents of free-text fields beyond a simple count of the number of times each full-field value occurs.  The new generation of data quality solutions provides users with the ability to analyse the contents of text fields and extract valuable knowledge from it.

Organisations that can understand dirty data and extract golden nuggets of information from it have the power to turn what was once viewed purely as a liability into a valuable asset.

The resurrection of Mr. Smith

"Nothing in life is certain except death and taxes" - Benjamin Franklin

The truth of the second of these is undeniable, but you could be forgiven for doubting the first if you worked in the branch office of some banks.  How would you react if, as a bank teller, your computer records showed that the customer standing in front of you supposedly died a year ago?  “Um,… how are you feeling today Mr. Smith, you’re looking a little pale?”

What leads to this situation is often a muddle of processes, and people using workarounds to beat the system.  For instance, I’ve discovered that a common practice in some banks is to flag a favoured customer as deceased so that they can close a savings account and withdraw money without a penalty.

In other cases the confusion has come as the result of genuine bereavement.  Rather than comply with documented procedures, following the death of a customer somebody has decided that it is more expedient to over-type the original customer’s details with the name of the person who is granted probate.  The one field that can’t be changed by anyone once it has been entered is the date of death; so there it sits, alongside someone else's details.

Given that this is such a sensitive topic, I am, on the one hand, astonished at how some people are willing to act so flippantly, but I also understand why people find these workarounds so useful.  The consequence of their action may be something that can be regarded as a data quality problem, but unless the inadequacies of the underlying processes are resolved, any fix of the data will not be sustainable.

User error

When The Data Warehousing Institute asked in a survey "where does dirty data come from?" the main cause cited was sloppy data entry.  But my experience is that it's sometimes unfair to blame the users; let me give you an example.

I was asked to look at some problem addresses for a UK-based client's data migration project.  The dodgy records were coming from the company's CRM system and the users entering the data were being blamed for the poor quality.  When I looked at the data, I spotted a trend - all of the information was there, just in the wrong order, so I asked to see the data entry screen.

I talked to some of the data entry staff, and watched them enter some new customer records.  Every record they entered looked fine; the addresses on the screen read perfectly.  The problem was the screen layout and the fields that they we putting the address into.

For some reason best known to the CRM system vendor, the address was represented as low-level elements, which appeared on the screen in a 2-column tabular format.  The data entry staff have no idea what a dependant thoroughfare or a double dependent locality are, so they simply entered the address as they would expect to see it on an envelope, using the fields in the left-hand column.

The problem was compounded by the fact that the fields weren't in the order that they occur in a correctly formatted address.  During the migration, the addresses were rebuilt, but this time they followed the Royal Mail's standards, in short the address was put back together in a different order.

So who should we blame for these data quality issues?  Should we put it down to "user error" or should be look to the people responsible for the poorly thought through, and over-engineered CRM system?