dq:view - Steve Tuck on Data Quality

dq:View has moved...

Hi folks,

Thanks to everyone that has been following dq:View, but I've now taken the decision to move the blog to Datanomic's corporate site. I hope you'll continue to follow my witterings at http://www.datanomic.com/category/resources/blog/ or via an appropriate feed; RSS or Atom.

You can also find me online on LinkedIn as http://www.linkedin.com/in/SteveTuck, on Plaxo as http://stevetuck.myplaxo.com and if you're into Twitter, you can follow my Tweets at http://twitter.com/SteveTuck.

All the best,

Steve

p.s. There are already 2 new entries for you to read on data quality related topics at http://www.datanomic.com/category/resources/blog/.

Trillium Software's Identity Crisis

They say that imitation is the highest form of flattery.  I thought someone was pulling my leg when I first heard about this, but it's true - Trillium Software is currently paying for an advertisement on Google, that uses one word only - Datanomic!  Why would such a well established data quality software vendor make such prominent use of a competitor's name?  And why has Trillium singled Datanomic out for this special treatment?  I'll let you make up your own mind about that.  Meanwhile, here's a screenshot I just grabbed that shows the advert.

---

---

Feel free to Google Datanomic and click on Trillium's link - it takes you to the registration page for a White Paper, but if you want the real Datanomic, simply go to www.datanomic.com.  And Kevin, well spotted but no, this doesn't mean that Datanomic has been acquired by Trillium Software!  LOL

CDI - is it just another TLA (three letter acronym)?

I sometimes wonder if there’s substance behind the new catchphrases that the IT industry loves to throw about.  Take CDI, or Customer Data Integration, for example; is this some new, bright, shiny technology?  Or is it the cynical re-branding of something we already thought we had?

There is considerable confusion between CDI and that much-loved TLA (Three Letter Acronym) CRM – Customer Relationship Management.  CDI promises a single, 360 degree view of the customer; didn’t CRM promise the same thing?  I don’t blame anyone for being confused as, at it’s inception, I can remember myself thinking of CRM as an approach rather than a piece of technology and the single view of the customer was at the heart of it.  The truth is that, whilst CRM may have postulated the vision of a single customer view, the CRM vendors failed absolutely to deliver it.

That’s because CRM vendors focussed on the customer contact end of things; producing call-centre and sales-force automation technology, but did nothing to address the management of all the data that they collected.  Sure, the data model at the heart of any of these systems may support a single customer view, but without the processes to manage the quality of the data and identify and prevent duplicate customers from being created.

Furthermore, the processes that have been built around many of these systems have often encouraged the creation of dirty data rather than prevented it.  The most common name in one of my client’s CRM database was . .. (that’s a forename of “dot” and a surname of “dot dot”).  No, they had not started working in Morse Code, they had just paid the price for measuring the performance of their customer contact centre purely on call volumes, rather than the quality of service.

So what is Customer Data Integration and why is it different.  Firstly, let me say what it isn’t – it isn’t a piece of technology, the silver-bullet to solve all of your customer data challenges.  CDI should be a business objective – the management and coordination of all the information that a company holds on its customers.  For some, this may mean pulling all of their data into a single database and managing it there; but they will be the rare exceptions.

The reality is that most organisations will continue to live with multiple systems, including CRM, but will seek to create and manage the single view across all of them.  This presents them with two clear options: one where the data is federated and the single view created on the fly; and the other (I suspect more popular) option of creating a centralised “hub” that holds the master customer data with “spokes” that connects it to each of the other systems and synchronises data across them.

Critical to any of these approaches is the management of the quality of the data.  You can spend a fortune on a new car, but it won’t get you very far if you fill it with sludgy fuel and run it without a fuel filter.  In business, the challenge of dirty data must be tackled throughout the enterprise.

So is CDI something new?  Well I’m afraid, in true “there’s nothing new under the sun” style, the answer is no (I delivered my first CDI solution more than a decade ago, although the term was not widely in use then).  My hope is that the technology vendors supports the business community in achieving CDI as an objective, rather than uses it as just another way to sell CRM.

Gone Away - The need for Patient Data Integration in the NHS

In my blog entry on 28th March (Lost in migration), I discussed the issues surrounding the migration of patient data at the Nuffield Orthopaedic Centre.  This is part of a national programme to bring together patient data from different NHS Trusts to provide a "single patient view".

Each trust has been given a timetable to complete a CRS Migration, moving data from their Care Record Systems (CRS) or Patient Administration Systems (PAS) to a central "spine".  The Nuffield got a lot of press attention because it was one of the first trusts to complete a migration and ,as an out-patient at the centre I took a personal interest in it.

The need for Patient Data Integration is obvious; of course every clinician I see should have a complete history of my health treatment.  But just how disconnected the present system is was brought home to me again recently, courtesy of a good old "Gone Away".

My consultant at the Nuffield has decided to refer me to another department, which is based at the John Radcliffe Hospital (also in Oxford) and duly sent a note through to the appropriate person.  Of course, the easiest way of identifying me in the NHS computer systems is by my NHS Number, so this was supplied.  Unfortunately though, the systems at the Nuffield and the JRH are disconnected, so my details are stored separately on each.  And, having not received treatment at the JRH for some years, they still had my old address - so that's where they sent my appointment.

Thankfully, the current occupiers of the property still had a copy of our forwarding address and the letter arrived just a few days later.  The impact in this case was minimal, but consider what happens if a treatment centre cannot identify a casualty because their computer system is disconnected and out of date.  The diligence of the health workers means that most cases are resolved, but it takes time, effort and yes, money to do so.  Joining up the hundreds of NHS computer systems will improve efficiency, but most of all it will improve patient care - that's why NPfIT, the National Programme for IT, needs to succeed.

dn:Director - a fresh approach to data quality

Why do so many organisations turn a blind-eye to data quality?  One thing for sure is that the legacy data quality software providers have done little to help address this crucial business issue by delivering products that require years of expertise to successfully leverage all of the functionality available (and, just as importantly, to know when to use something else instead).  After a dozen years of working in the field, and having built a highly profitable consultancy business to help clients address this short-fall, I decided a year or so ago to join Datanomic.  I'm delighted to say that, last month, we celebrated the launch of dn:Director, a data quality product that is setting new standards for data quality management in the 21st Century.

I've been privileged to work on data quality projects with many leading, blue-chip companies over the years, but one of the things that struck me was that I was being asked the same questions by clients in 2004 as I was asking myself more than a decade earlier; they were identifying the same old deficiencies in data quality products and having to employ the same workarounds to resolve them.  Sure, the vendors have done something to smarten up the look of their software, but, under the covers sits essentially the same code that was initially developed for mail-room efficiency in the 1980's.

Two more things struck me:

1. All of the software vendors talked about delivering a tool for "business users" but the reality was that just about every project relied on the IT department to develop the business rules.
2. Because of the complexity of using the software to good effect, the cost and duration of projects was prohibitive; the reason I was working with so many blue-chip companies was that they were the only ones that could afford to undertake such major projects!

These were the things that motivated me to create Tranato and subsequently to join Datanomic in 2005 and bring together the two technologies under a shared approach.  Put simply, we feel that a data quality product needs to be much more accessible - you shouldn't need to be a software guru to get value from it.

dn:Director is the result of many years experience in data quality and data management; not just my own, but that of people like Gerry Kelley (Datanomic's VP of Professional Services) and his team, and the shared experiences of our clients and partners.  Taking Datanomic's approach (The Four Cornerstones) and methodology as its foundations, dn:Director has been built from the ground up, using the best-available modern technology.

Developing dn:Director in Java and using standards-based interfaces (such as JDBC, JMS and XML) has enabled us to deliver a technically advanced and extensible data quality product that supports both batch and real-time processes (providing data quality services through SOA).  But the thing that everybody notices first is just how easy it is to use - you should hear what out customers and partners have had to say about it:

"This is great - it's so easy understand and configure business rules"

"I love the way that you can build rules from the data - it's so quick and intuitive"

"This will halve the time it takes to deliver a project"

For more information visit Datanomic's website or call on +44 (0)1223 228400.

Note: I know this is very commercial for a blog entry, but given the amount of personal time, energy (and money) I've committed to making dn:Director a success, I hope you'll forgive me.

Retailers need to clean up

The Global Data Synchronisation (GDS) Network is an Internet-based supply chain initiative that was founded by two international standards groups; UCC (Uniform Code Council) and EAN International.  Industry heavyweights, including Tesco, Asda Wallmart, Proctor & Gamble, Unilever, Cadbury Trebor Bassett and Kraft, are among the retailers and suppliers calling for other industry players to sign up to the GDS Network.  However, if so many companies are to synchronise their data, it is crucial that it is clean, accurate and up-to-date in order for GDS to work effectively.

Currently, the information which appears when products are scanned differs according to the retailer, but GDS would mean the same standardised data is used on all products, which should cut supply costs by millions.  Standardising product codes across the whole supply chain has significant advantages – the relationship between the supplier and the stores would be streamlined, resulting in faster delivery times, better stock control and improved reporting on sales and revenue.

With almost 50% of the UK grocery trade expected to adopt the network within months, this represents an extraordinary opportunity.  However, retailers must ensure that the data within their own organisation is correctly aligned in order to benefit from GDS benefits further down the line.

Retailers need to carry out a systematic audit to establish exactly where any discrepancies, omissions or duplications in their data lie.  Then the information should be cleaned and consolidated before it is used if the anticipated benefits of GDS are to be realised.