Of Data Mining and its Challenges

 

BY FAR THE MOST DIFFICULT TASK since I started to work as a consultant is to extract RELIABLE data from the client. A typical engagement would start with a kick’-off meeting with your well-groomed team of consultants and the client’s people. All ready to tackle the problem that your firm has been hired to figure out. Because you have been thoroughly trained in Consulting 101, you hypothesized about the problem, broke down the issue into pieces and re-arrange everything into a cohesive, logical structure. Pfiu, that was the hard part, now let’s just get some data to confront our hypothesis and we can keep the ball rolling. Well, bad news. The “mundane” task of gathering data is not easy. It is not even mildly difficult. It is VERY HARD.

What I am talking about here is of course of data good quality. Information that comes from trustable source, that is updated frequently and that does not exist in different versions. And this, my friends, is very hard to obtain for a very simple reason. Most clients are not data specialists. They know how to run their business based on their knowledge and experience, helped with common metrics and financial statements. But YOU, as a management consultant, are asked to look at a complex problem (hint: if the problem was easy, you wouldn’t work for this client), using new type of data broken down by new dimensions and residing in different business units. Yes, you are in trouble.

It is becoming a cliché nowadays to say that “information is the most valuable commodity in the 21st century” and that every company in its right mind should harness the knowledge and technology to productively use this data. Well, this truism has always been true to consultants. After all, that’s what we have been paid for, ever since the profession emerged: interpret data and make it useful! If data is the new gold, then consultants are the early prospectors.

All would be well indeed, if data was plentiful and of sufficient quality. Unfortunately, must like in the Old West, some sites are  well furnished in raw material while some are frankly nothing more but scorched land. You would be surprised to know how many clients actually run data deserts.

This is sometimes even more frustrating as you may think that technology, for once, is a true ally. After all, when a company decides to implement an ERP system, shouldn’t a lot of good data be generated and easily accessed? Well, I will give you a typical example from a real engagement, illustrating that sadly a big nice IT system may not help you at all, but can even work against you. Of course, names and specific facts are altered for confidentiality reasons.

We were asked to forecast the demand of a big client with operations all over South East Asia, who just rolled on the implementation of a major ERP system in all its regional divisions. To start with, the client did not have any demand data, but only shipment-based sales data (if you can’t see the difference between the two, check out my other blog, I will talk about it very soon). Deciding to use shipment-sales as a proxy for demand, we aimed to collect the data for every regional centres. Something that, on the surface looks quite easy. By collecting data from every country, it appeared quickly that the sum of the regions did not match with the official corporate figures, even though all the data came from the same ERP. What we realized, is that a lot of cross-selling was happening between the different countries, something that was not envisioned by the ERP system. So in order to get the “real” figure, the data would need to be cleansed, using manually recorded transactions that started to vanish as soon as the new system started to be implemented.

Challenges like the one presented in my little anecdote happen ALL THE TIME on an engagement. You actually demonstrate a lot of your consulting skills by being able to extract good quality data in a minimum of time. Victor Cheng, from caseinterview,com, mentioned in one of its articles that contrary to common business school teaching, strategy is easy and implementation is hard. I believe this really materializes when you must pull out the data to validate all your smart business conjectures. Be ready to sharpen your pickax, you have a lot of material to digg out now.