The benchmark problem
AI benchmarks are designed to be standardized, reproducible, and comparable. They succeed at all three. The problem is that none of those properties guarantee relevance to your use case. A model that tops a leaderboard on general reasoning may still fail at the specific task your organization depends on.
Why benchmarks diverge from reality
- Distribution shift — benchmark data is curated; your data is messy.
- Contamination — models may have seen the benchmark during training, inflating scores.
- Single-metric blindness — a benchmark rewards one dimension and ignores everything else that matters.
What to measure instead
Build an evaluation set that reflects your actual workflow:
- Collect real examples — pull 100-200 inputs that represent the tasks you actually need done.
- Define success concretely — what does a good answer look like for each one? Write it down.
- Score blind — evaluate model outputs without knowing which model produced them.
- Track failure modes — not just accuracy, but the kinds of mistakes. A model that is wrong in predictable ways is more trustworthy than one that is wrong in surprising ways.
The principle
A benchmark tells you what a model can do in general. Only your own evaluation tells you what it will do for you. The investment in building that evaluation pays off every time you consider a new model, a new version, or a new prompt.
Knowing quicker means measuring the thing that matters, not the thing that is easy to measure.