Here’s a habit worth building: the number that confirms what you already hoped is the one you audit hardest, not the one you accept fastest. Nobody opens an investigation into good news. That is exactly why good news is where the errors hide.
We ran into this ourselves, two weeks before opening Kaarya’s first public test.
The question we needed answered was narrow and important: of the people who sign up, what percentage actually connect a WhatsApp number? Kaarya’s whole premise is that a team can run its project tracking from WhatsApp. If people sign up and never link a number, the product doesn’t have a UX problem, it has a reason-to-exist problem. Better to find that out in a fortnight of testing than a year into building on top of it.
So we built a funnel. Numerator: count of verified WhatsApp links. Denominator: count of signups. Divide, watch the percentage, worry if it’s low.
The bug was in the numerator. It counted every verified link in the entire database — not just the links belonging to people who’d signed up through the test. That included the team’s own numbers from development, and every pre-existing internal account we’d used to build and check the feature along the way. The denominator, meanwhile, was scoped correctly to test signups only. Two different populations, one ratio.
Run that math with zero testers signed up and see what happens: denominator is effectively as small as the test cohort gets, numerator is however many internal links happen to exist, and the dashboard proudly reports 100% conversion. Before a single outside person had touched the product. The number wasn’t just wrong, it was wrong in the most flattering possible direction, which is precisely the direction I’ve learned not to trust.
That’s the reflex worth building: if a number looks too clean, check whether it’s actually measuring the population you think it is, not a bigger or friendlier one that happens to share a name. A metric that balances too easily is one you haven’t read closely enough.
We didn’t catch this by staring at the conversion number and doubting it — on its own, 100% just looks like a great day. We caught it because two different views of the same thing disagreed. The summary dashboard said one WhatsApp number was verified. The tester list, which showed individual testers and what each of them had done, said our one actual tester hadn’t verified anything. Both statements couldn’t be true. That contradiction is what sent us back into the query.
The fix, once we saw it, was almost embarrassingly small: the numerator has to be drawn from the same population as the denominator. Count verified links among the people you’re counting as signups, not verified links anywhere in the system. One clause. It is the kind of thing that is completely obvious stated out loud and was completely invisible while we were writing the query, because we were thinking about “verified links” as a fact about the product, not a fact about a specific set of rows.
The broader lesson is the one I keep relearning: a metric that tells you what you hoped to hear deserves more scrutiny than one that alarms you, not less. A bad number gets investigated by default — someone will ask why it’s bad. A good number just gets believed. If we hadn’t happened to build a second view of the same data, one with a different scope and a different shape, we might have carried that 100% into the actual test and only noticed something was off when the two dashboards disagreed in a way we couldn’t ignore any longer.
We fixed the query, matched the populations, and let the number tell us whatever it was actually going to tell us. That’s still the standard we’re holding ourselves to before we call anything measured: not “does the number look good,” but “have I checked what it’s actually counting.”