If you wanted to program a computer in the early 1980s, you didn’t have the option of coding using a keyboard. You had to create a series of cards, each punched with a pattern of holes. The holes had to be entirely accurate, in both pattern and placement. A typical card contained hundreds of potential positions. Just one error in your card deck would cause the entire card to be invalid.
In the early days of office computing, mistakes were simply not an option. There was zero tolerance. Correcting errors, or repunching jammed cards that the machine didn’t verify, could take several days, per card.
Nowadays, we’re so used to getting instant results that we’ve become far more error tolerant, and we don’t have this perfectionist approach. We can add a record to a database in well under a minute, and we have ways to get around pesky validation errors when a record won’t save.
If there’s already a record of a person in the database, we can always add the word ‘NEW’ to their name, rather than backtracking and looking for the duplicate.
What harm can it do, really?
As we move towards an age of complete automation, fudging verification and tolerating bad spelling is starting to hinder our success, and our profitability. Perfect data is rarely affordable, or achievable. But we are certainly becoming increasingly aware of mistakes, because it’s stopping us working as efficiently as we need to.
Punch cards aside, if you aren’t putting the right effort into accuracy, there are three main cases to answer:
Put all of this into the context of automated working, and we have a recipe for disaster. One bad record in a good database is going to filter through into every other system. Every department will be inconvenienced. Everyone is going to waste time. When it comes to pulling together another report, you won’t be able to trust even one of them.
There are other reasons to focus on data quality , quite apart from the need to ensure profits and reduce waste. Consider the market for wearables. We’re already seeing these devices being used as evidence in court.
One example is the very serious issue of a rape case reported in Lancaster, Pennsylvania in the US. When investigating the data reported by the woman’s Fitbit fitness tracker, detectives found that her movements did not support her story. The Engadget report makes reference to the fact that wearable data is never totally accurate, which is a useful reminder of the dangers we face in putting too much trust in faulty statistics.
There are also implications for the many organisations that access anonymous data. In a recent survey by KPMG, 78 per cent of respondents said they’d be happy to share wearable data with their GP. This could have a direct impact on healthcare outcomes for individuals, and we could reach a stage where devices like this are informing healthcare policy and planning.
If we’re going to start using data in this way, we need to be absolutely sure it’s correct. The rape case is a rather extreme example of this, and the data was presumably analysed in context. But it’s a timely reminder that data quality can no longer be considered optional in any case.
It’s very difficult to put an absolute figure on the cost of poor data, and businesses need to take a balanced approach when seeking data quality solutions. There’s always a tipping point for data quality, where the investment makes a worthwhile difference without bankrupting the business.
But for the purposes of this article, we need to look at the cost of inaction, as well as the cost of change, including:
So yes: transformation, automation and modernisation all cost money, and nobody likes to spend. Retraining requires investment, and increasing the quality hit rate is as much about your staff as your systems.
But inaction makes data unfit for purpose.
So is data quality an optional extra for a modern business? We’d argue that it’s not – it’s an essential, core component. And businesses are going to have to adapt to survive. You can’t deliver an exceptional experience to customers if you’re not sure who they are. And you cannot make a positive change to the quality of data without changing your processes – and your mindset.