User error

When The Data Warehousing Institute asked in a survey "where does dirty data come from?" the main cause cited was sloppy data entry.  But my experience is that it's sometimes unfair to blame the users; let me give you an example.

I was asked to look at some problem addresses for a UK-based client's data migration project.  The dodgy records were coming from the company's CRM system and the users entering the data were being blamed for the poor quality.  When I looked at the data, I spotted a trend - all of the information was there, just in the wrong order, so I asked to see the data entry screen.

I talked to some of the data entry staff, and watched them enter some new customer records.  Every record they entered looked fine; the addresses on the screen read perfectly.  The problem was the screen layout and the fields that they we putting the address into.

For some reason best known to the CRM system vendor, the address was represented as low-level elements, which appeared on the screen in a 2-column tabular format.  The data entry staff have no idea what a dependant thoroughfare or a double dependent locality are, so they simply entered the address as they would expect to see it on an envelope, using the fields in the left-hand column.

The problem was compounded by the fact that the fields weren't in the order that they occur in a correctly formatted address.  During the migration, the addresses were rebuilt, but this time they followed the Royal Mail's standards, in short the address was put back together in a different order.

So who should we blame for these data quality issues?  Should we put it down to "user error" or should be look to the people responsible for the poorly thought through, and over-engineered CRM system?