“We need to fix this bug in the software, right now!” Back when I ran operations at a midsize software company, I recall the day our top sales person burst into my office with this passionate plea. Which she immediately followed up with, “But it’s OK, I’ve spoken to the engineers and they’ve added it to the priority list.”
“Slow down a bit,” I responded, “what bug, and why the urgency?”
She described the bug, which I was aware of as it was on the resolution list for the next major release. Which she followed with “It needs to be fixed. It's stopping customers renewing.”
“How do you know?” I asked.
“I just got off the phone with ABC Inc. and they said it’s causing them a major problem, and they won’t renew their contract until it's fixed.”
“OK. So is it stopping them working, and have they filed a support ticket?”
“Don’t think so.”
“And how many people at that company is it affecting?”
“The guy I was talking to didn’t say.”
“And how many renewal seats are we looking at — what’s the revenue impact?”
She threw out a number that was probably mid-range of what was in her pipeline.
“Let me look into it.” It only took a short walk around the office to discover that the company concerned hadn’t logged a support call, nor had any of the other sales folks heard anything similar from their accounts. Grabbing a couple of coffees I wandered into the sales office, handed my panicked sales person a fresh jolt of caffeine and summarized my findings. “So basically you just asked engineering to change their schedule because you had a call with a frustrated user at a mid-size client? All on the basis of a data point of one?”
“I guess so.” She responded a little sheepishly. “But he had a problem.”
“Yes, and I’ve had support give him a call and walk him through the issue. It’s great that you are concerned about a client, but before we invest time, effort, and money in making changes to our development program we need to make sure that the need applies to more than one customer.”
This was one of my most vivid memories of the dangers of having a “data point of one,” but it’s always stayed with me, and I’ve used that phrase “data point of one” many times since.
Related Article: Is Your Company Data-Driven or Data-Informed
You'll Need Data, Context and Understanding
All too often I’ve seen people respond to something from an instinctive and emotional perspective without thinking through the data, or as in the case above, the lack of data.
At management school the idea of taking the emotion out of a situation and focusing on the data was an oft-repeated piece of advice. It’s a good one. However emotions can cloud the data too, especially when you only get presented with the data to back up a certain need or a singular point of view.
Even so data on its own doesn’t always tell the story, it needs context. Take these data points for instance: 56.1, 53.2, 50.9, 51.1, 49.8 — these were very important to me over the last weekend.
I occasionally compete in a type of competitive driving event known as Autocross. It’s a timed event, and those were my run times, in seconds, on Sunday. As you can see my times got better each time out, except for my fourth run. The data says I went slower than the previous run, but it doesn’t say why.
At the start of the fourth run there was a change in the starting marshal who had a slightly different way of signaling. It was only a small thing, but it was enough to distract me which led me to be slower getting off the start line. While the figures show an overall improvement in my performance on the day, it needs context to understand the change in the progression of the individual data point.
Raw numbers can be informative, but we need to know more. When we make data-driven decisions we still need to consider the human factors.
- Is there an agenda driving what data is being shared, and the way it’s being presented?
- Are there multiple ways of interpreting that data?
- Are there other external factors at play?
- What’s the story behind the data?
Data for data’s sake is a useful tool, but it doesn’t paint the full picture — that needs a combination of data, context and understanding.