Address data is semi-structured, making it one of the most challenging components in a data matching activity. For long now, manual data matching methods including extensive SQL programming and spreadsheet formulas have been used to match address lists. While this may have been workable and effective in the past, it is no longer a viable method to handle complex data from third-party sources. 

In this quick post, I cover key challenges to manual address data matching and how a…

Mergers & acquisitions happen when companies believe they are more valuable together than when operating separately. The companies join workforces, systems, infrastructure, and data to become a new, more powerful, more valuable, more effective entity. That is only until they realized they overlooked or underestimated the key issues with data, IT infrastructure & integration plans. In fact, most merger and acquisition plans fail miserably because of data…

What’s the #1 enemy to profitable machine learning, AI & data-driven initiatives?

Dirty data.

And who has had data on their agenda for a decade but are still not deeply involved in data projects?

The CEO.

In a 2019 Deloitte survey, 63% of…

Who are our Data Quality Heroes?

Lemahieu W., vanden Broucke S., Baesens B.

This article is based upon our upcoming book Principles of Database Management: The Practical Guide to Storing, Managing and Analyzing Big and Small Data, www.pdbmbook.com  See also our corresponding…

Although I deal with many different types of metrics, I believe they can be generally classified as follows: 1) time use; 2) alignment; 3) production; 4) performance; 5) service; 6) and market.  In this blog, I will be providing some comments pertaining to each.  Although I have yet to encounter any myself, I am certain that there must be text books on the issue of operational metrics and how to make use of them.  However, I personally developed nearly all of those that I use.  Although I do…

In this article I explore some of the key concepts of data quality management and how to build a strategy for continuous improvement. I won’t be covering every possible scenario, process, method or problem; only those that are common across most industries and those that have proved useful on my own personal journey.

Hopefully, we already agree that good data quality is an essential part of business intelligence and a foundation on which you build your systems, processes and…

I have written in the past about the difference between market demand and operational capacity - and how difficult it is to determine what exactly is being measured in relation to either.  Has the demand for a product declined, or is the organization simply less capable of satisfying it?  For example, the fact there are no bananas in the grocery store does not mean that there is no demand for bananas; but the absence of revenues from the sale of bananas might be regarded, rather erroneously,…

Creating a data management strategy doesn’t have to be difficult. By understanding how your data flows through your business and teaming it with an intelligent management solution you can ensure your business’ longevity.

Every business collects data; however, many do not manage their data effectively and often have no data strategy in…

Do you often go with gut feeling rather than data and insights? Is your data stored in separate databases, in different formats with different values? We all have bad habits and some are a little hard to kick. However, if there is one you must break, it is surely to make your bad data habits a thing of the…

This blog is about the lived experience of "becoming a statistic." In our increasingly data-oriented society, the use of data serves not merely to try to understand but also to control people and determine their placement in the scheme of things. Central to the theme of systemic racism, gender discrimination, and the formation of class strata is the elevation evidentiary artefacts such as statistics in order to structurally define and diminish "individuals." The purpose is to make lesser the…

I found some leftover hamburgers in the fridge. I decided to stack a couple of them together to form a colossal “super-burger.” At the time, I didn’t appreciate how doing so would make it almost impossible to physically fit the burger in my mouth. I squished and squeezed the burger until it was flat enough to eat. Such are the problems of physics that become…

The idea and concept of “Big Data” has been around for a while now, however it seems there are many people who still believe it is shrouded in mystery. Therefore, in the following post we are going to debunk some of the top big data misconceptions.

Every business has access to data,…

If you wanted to program a computer in the early 1980s, you didn’t have the option of coding using a keyboard. You had to create a series of cards, each punched with a pattern of holes. The holes had to be entirely accurate, in both pattern and placement. A typical card contained hundreds of potential positions. Just one error in your card deck would cause…

Whatever your business sector, data is your most valuable asset. Along with the machinery and stock you hold, data and insights hold the key to profit and growth. But it has the unique ability to unite every department, and every function. It can reveal problems in processes, drive productivity among your staff and ensure everyone is ‘singing from the same hymn…

So far in 2016, we’ve heard a lot about Big Data, the Internet of Things, and the hottest new marketing technique on the block: ‘hyper-personalised’ marketing. Marketers are increasingly interested in focusing their efforts on individuals, not database segments, because that all-important personal attention is shown to increase engagement, inspire customer loyalty and…

For ever it seems, we’ve been warning about the dangers of low quality data. Our warnings have been reinforced and echoed by some of the world’s biggest think tanks. However, despite this, some organisations still haven’t acted to improve the quality of their data.  And we’re wondering why?

Over the last 12 months, we’ve blogged about business automation , and about…

The company that made your smartphone is no longer a hardware manufacturer. It has become adept atharvesting and managing data from it users, too. Every device has a series of sensors, as well as a user…

I find that different types of surveys represent a large source of data for many organizations: client questionnaires; recruitment interviews; incident debriefings; interrogations; borehole drilling surveys; quality control checks; marketing surveys; security and patrol logs; and inventory audits. I believe that for many people, the idea of collecting information using surveys makes sense; and they recognize the need for the data. Problems arise in relation to the transition from survey to…

All businesses are at the mercy of data quality challenges. From the moment you capture your first lead, you’ll be fighting a battle against data decay. The bigger the database gets, the more problems the business can encounter, and it isn’t easy to single out a…