Your model comparison is measuring the wrong number.

Every price list is per token. Your bill is per finished piece of work. Those two numbers can rank models in opposite orders.

Two numbers that disagree

Provider pricing pages are all denominated the same way: so much per million input tokens, so much per million output tokens. It is a fair unit for the provider, because tokens are what they process. It is a poor unit for you, because tokens are not what you are buying. You are buying completed work.

The gap between those shows up the moment a task takes more than one attempt. If a cheaper model needs two passes and a validation retry where a more expensive one succeeds immediately, the cheaper model has just cost you three calls plus the engineering time to handle the failure path. The per-token price list ranked them one way. Your invoice ranks them the other.

Where the extra tokens hide

Three things inflate real cost above the sticker price, and none of them appear on a pricing page.

Retries are the obvious one. Any task with a validation step, structured output that must parse, code that must compile, an answer that must cite a real source, has a failure rate, and every failure is a call you paid for and threw away.

Reasoning tokens are the second. A model that thinks before answering generates tokens you never see but are billed for as output. A short visible answer can carry a much larger output bill than its length suggests.

Replayed context is the third, and it is the one that grows. Every turn where you resend conversation history is history you pay for again. A conversation at turn fifty costs far more per turn than the same conversation at turn five, even though nothing about the model changed.

What the current generation is claiming

This is why the framing in recent model releases has shifted. Anthropic describes Claude Opus 5 as more than doubling Opus 4.8's performance on Frontier-Bench v0.1 at a lower cost per task, and as landing within 0.5% of the top CursorBench 3.2 score at roughly half the cost per task.

Read carefully, those are not per-token claims. They are claims about the total cost of getting a job finished, which is the honest way to compare and also the harder thing to verify. Treat a vendor's cost-per-task figure the way you would any vendor benchmark: as a hypothesis to test on your own workload, not a settled result.

Working out your own number

The measurement is not complicated. Pick one task your product performs often. Run it a hundred times against each candidate model. Record total tokens spent including every failed attempt, and record the success rate against whatever "correct" means for that task.

Divide total cost by successful completions. That is your cost per task, and it is the only number that predicts your bill. It is common for the ranking it produces to differ from the ranking on the pricing page, which is precisely why it is worth an afternoon.

What to do

Pick your highest-volume task and define what a successful completion means, precisely enough to measure automatically.

Run a hundred trials per candidate model, counting tokens from every attempt including failures.

Divide total spend by successful completions. Compare that number, not the pricing page.

Instrument retry rate in production. A rising retry rate is a cost increase that no price change will explain.

Re-run the comparison when you change the prompt. Prompt changes move success rate, and success rate moves cost per task more than token price does.