Data Quality Control vs Data Quality Assurance
In a recent blog post on data quality (rather the lack of it) Charles Burleigh asked the question, “Should we treat the cause or the symptom?”.
My response to Charles was:
“For as long as people in data quality continue to say “it is almost impossible to guarantee the quality of your data” it will remain so.
Not so long ago people in industry were saying “it is almost impossible to produce products with zero defects”.
It was not until Quality Assurance came along and people started asking industry leaders the question, “Why do you spend time and money turning out products with some unknown number of defects and then spend more time and money trying to find them?”. Note: Finding these defects was given the paradoxical name of “Quality Control”!!
Many of the industry leaders replied, like data quality practitioners do today, that it was impossible not to do so.
But then some manufacturers (primarily in Japan to begin with) embraced Quality Assurance started to do the “impossible” and produce products with zero defects!
Now zero defects it is a fact of life.
So I ask business leaders, “Why would you feed defective an erroneous data into your systems and then spend time and money trying to find and remove it?”
We have to move out of the era of Data Quality Control (which deals with find defects) to the era of Data Quality Assurance – which prevents occurring.”
What do you think? Please leave a comment below.
(You might have to click on the “No Comments” link to open the comments box)
3 comments
I remember a joke from the 90’s.
“An American company wanted a Japanese factory to produce floppy disks.
The requirement for quality was: 3 out of 10.000 with defect.
When the Japanese factory made first delivery a note was attached:
Here are the first 10.000 floppy disks. We have packed the 3 with defects separately.”
And yes John, we are working on upstream prevention.
“We have to move out of the era of Data Quality Control (which deals with find defects) to the era of Data Quality Assurance – which prevents occurring.”
I wish the world of Data Quality could look like this but in my world (The Nordics) most of the organizations I meet have not reached the level of maturity needed to understand that data defect prevention can only be performed efficiently if you understand the problems first. This means identifying the problems first before engaging in any preventing measures. Recent surveys have found that organizations rarely measure their data quality. So how will these organizations be able to prevent the defects without knowing the severity of them?
So I agree fully that the bucket on the floor will not help you in fixing the leak in the roof but it will sure help you to keep the floor dry while you are repairing the roof correctly.
While “zero data defects” sounds a little pie-in-the-sky, I can’t argue that it isn’t worth striving for. Taking steps to prevent data of poor quality from entering a system in the first place is worth the effort.
Leave a Comment