Clustering Illusion

The Clustering Illusion is a cognitive bias that occurs when we perceive patterns in random sequences of data or events, even when there's no actual correlation or causal relationship present. This bias reflects our brain's tendency to seek order in randomness.

---

Two preeminent scholars on the clustering illusion, psychologists Daniel Kahneman and Amos Tversky, assert that the representativeness heuristic causes the clustering illusion, a cognitive shortcut where a small sample of data is assumed to be representative of the entire population that it’s pulled from.

The human brain's ability to recognize patterns and draw conclusions is truly remarkable. However, remember to be cautious when using small sets of data to make assumptions about larger populations.

---

I want to tell you a story about opportunity costs.

But first, let's hear from this week's sponsor...

---

---

Now back to the story ⏬

We’ve all had stubborn stakeholders, right?

I think there’s a reason stakeholders tend to get a bad rap, unfortunately.

Here’s a story about how the clustering illusion wasted a year of work and lots of money building things no one wanted.

Some time ago, I worked on a team with a stakeholder who tended to be somewhat stubborn. Nice enough guy, but when he got something in his head, good luck convincing him otherwise.

We were building software with a pretty diverse user base. There were large customers with thousands of users and small customers with a few hundred. There were even lots of really small customers with only a handful of users.

As is usually the case, the big-name, larger customers get direct access to high-level executives to voice whatever concerns they have.

Unfortunately, the squeaky wheel tends to get the grease.

The trouble is these large, big-name customers, are actually a smaller overall fraction of the total user base.

But they tend to be the loudest and they tend to be the ones that most of the higher-ups listened to.

You can probably guess what happened next.

Our stubborn stakeholder had a nice long chat with someone from one of these large, well-known customers. They had a feature they wanted built and were insistent that it was business-critical for them.

This stakeholder, armed with this information and a bit of confirmation bias, wanted to find out if the other, larger also well-known customers also needed this feature.

Turns out they did!

... Or at least they said they did.

Fast forward a year and god knows how much money spent, and guess what?

Only a handful of users were using the feature.

Doing some interviews we found most customers had no use for this feature.

---

In hindsight, this stakeholder talked to only a handful of specific customers and heard what he wanted to hear.

In general, it wasn’t a bad thing that we gave this larger customer what they wanted. In truth, they were the only people using this specific feature.

It didn’t kill the business, but there was certainly an opportunity cost here.

What else could we have been working on that would have brought more value to the software overall?

We’ll never know, because we didn’t do the research to find out.

🤷

---

🎯 Here are some key takeaways

1️⃣ Be cautious of pattern-seeking: Recognize that our brains naturally seek patterns and structure, even in random data. Always validate observations with rigorous analysis before drawing conclusions.

2️⃣ Rely on statistical significance: Use appropriate statistical methods to determine whether observed patterns are statistically significant or simply the result of chance.

3️⃣ Gather more data: The larger the dataset, the more reliable the analysis. Collecting more data helps reduce the influence of random variations and provides a more accurate picture of what is going on.

4️⃣ Pair Quant with Qual: Complement quantitative data with qualitative insights through surveys, interviews, ethnography, or usability studies. This will help you gain deeper insights into the problem. Find the “why” behind the “what.”

5️⃣ Communicate uncertainty: When presenting data or insights to the team, communicate the possibility of random variations and the potential for the clustering illusion to avoid drawing misleading conclusions.

Learn more about the Clustering Illusion!​

---

---

---