Skip to content
craze_Platform
← Research

Method · 8 min read

Confidence, and how it is measured

Every result carries a number saying how far to trust it. Here is where that number comes from.

Confidence on a run is not a model’s self-report. It is derived from how that population has performed against humans it had never seen. A model asked whether it is right will usually say yes; the only honest way to know is to check it against people.

Held-out validation

A slice of every panel is withheld. Those people answer real questions, the population answers the same ones, and the two distributions are compared. The comparison is on the whole shape, not just the leading option, so a population that picks the right winner for the wrong reasons still scores badly.

Right winner, wrong shape
Real people
Stay 50% · Switch 30% · Stop buying 20%
Population A
Stay 52% · Switch 29% · Stop buying 19%
Population B
Stay 80% · Switch 15% · Stop buying 5%
Both populations pick “Stay” as the winner. Only A would have told you that one in five buyers is at risk, so only A scores well. Illustrative figures.

Why alignment is never one hundred

Ask the same person the same question twice a fortnight apart and they will not perfectly agree with themselves. Human test-retest sets the ceiling, and a population that claimed to beat it would be reporting a bug rather than a result. Alignment is best read as a share of that ceiling: how close the population gets to being as consistent with people as people are with themselves.

From validation to a number on your run

Validation runs every week, across subpopulations and live use cases. Those results train a confidence model that looks at your run and predicts how accurate it is likely to be, based on how this population has done on questions like yours. Your question is never assumed to be as easy as the average one.

  • The populationHow well it has aligned overall, and how much of it rests on a verified panel.
  • The kind of questionPrice and choice questions validate differently from awareness or attitude questions.
  • The segmentThin or unusual segments carry less evidence and earn less confidence.
  • The distance from past checksA scenario close to ones already validated scores higher than a novel one.

How to use the number

  1. 01HighTrust the direction, the ranking and the rough magnitude. Act on it.
  2. 02ModerateTrust the direction and the ranking. Treat the gaps between close options as unresolved.
  3. 03Medium or lowTreat it as a hypothesis worth testing, not an answer. Usually a sign the population needs more grounding.

When confidence comes back low

Low confidence is information, not failure. It usually points at one of three fixes.

  • Ground the populationAdd a verified panel base, or your own customer data, so the twins have more of the real behaviour to draw on.
  • Narrow the questionA scenario that asks about a whole category over a year is harder to validate than one decision at one moment.
  • Check it with peopleRun a small real study on the one question that matters most. Its answer also feeds back into validation, so the next run on similar questions starts from more evidence.

We publish validation results weekly. The confidence number on your run is only worth trusting because the record behind it is open to inspection.