Your model comparison is measuring the wrong number.
Every price list is per token. Your bill is per finished piece of work. Those two numbers can rank models in opposite orders.
- What providers publish
- Cost per million input and output tokens
- What you actually pay for
- Tokens across every attempt until the task succeeds
- Hidden multiplier
- Retries, reasoning tokens, and replayed context
- Worked example
- Anthropic reports Opus 5 beating Opus 4.8 at lower cost per task
- Practical rule
- A pricier model that succeeds first time can be the cheaper one
Two numbers that disagree
Provider pricing pages are all denominated the same way: so much per million input tokens, so much per million output tokens. It is a fair unit for the provider, because tokens are what they process. It is a poor unit for you, because tokens are not what you are buying. You are buying completed work.
The gap between those shows up the moment a task takes more than one attempt. If a cheaper model needs two passes and a validation retry where a more expensive one succeeds immediately, the cheaper model has just cost you three calls plus the engineering time to handle the failure path. The per-token price list ranked them one way. Your invoice ranks them the other.
Where the extra tokens hide
Three things inflate real cost above the sticker price, and none of them appear on a pricing page.
Retries are the obvious one. Any task with a validation step, structured output that must parse, code that must compile, an answer that must cite a real source, has a failure rate, and every failure is a call you paid for and threw away.
Reasoning tokens are the second. A model that thinks before answering generates tokens you never see but are billed for as output. A short visible answer can carry a much larger output bill than its length suggests.
Replayed context is the third, and it is the one that grows. Every turn where you resend conversation history is history you pay for again. A conversation at turn fifty costs far more per turn than the same conversation at turn five, even though nothing about the model changed.
What the current generation is claiming
This is why the framing in recent model releases has shifted. Anthropic describes Claude Opus 5 as more than doubling Opus 4.8's performance on Frontier-Bench v0.1 at a lower cost per task, and as landing within 0.5% of the top CursorBench 3.2 score at roughly half the cost per task.
Read carefully, those are not per-token claims. They are claims about the total cost of getting a job finished, which is the honest way to compare and also the harder thing to verify. Treat a vendor's cost-per-task figure the way you would any vendor benchmark: as a hypothesis to test on your own workload, not a settled result.
Working out your own number
The measurement is not complicated. Pick one task your product performs often. Run it a hundred times against each candidate model. Record total tokens spent including every failed attempt, and record the success rate against whatever "correct" means for that task.
Divide total cost by successful completions. That is your cost per task, and it is the only number that predicts your bill. It is common for the ranking it produces to differ from the ranking on the pricing page, which is precisely why it is worth an afternoon.
What to do
- 1Pick your highest-volume task and define what a successful completion means, precisely enough to measure automatically.
- 2Run a hundred trials per candidate model, counting tokens from every attempt including failures.
- 3Divide total spend by successful completions. Compare that number, not the pricing page.
- 4Instrument retry rate in production. A rising retry rate is a cost increase that no price change will explain.
- 5Re-run the comparison when you change the prompt. Prompt changes move success rate, and success rate moves cost per task more than token price does.
Common questions
What is cost per task?
Total spend divided by successfully completed jobs, counting the tokens burned on failed attempts. It captures retries, reasoning tokens, and replayed context, none of which appear on a per-token price list.
Can a more expensive model be cheaper?
Yes, and it is common. A model with a higher per-token price that succeeds on the first attempt can cost less per finished task than a cheaper model that needs two or three tries.
Why do reasoning models cost more than their output length suggests?
They generate internal reasoning tokens before the visible answer, and those are billed as output. A brief response can carry a substantial output charge.
How many trials do I need?
A hundred per model is usually enough to separate candidates on a task with a meaningful failure rate. If the models are close, the difference probably will not decide your bill anyway.
Sources
Read against the primary documentation rather than secondary coverage. Where a figure comes from a provider's own docs, it is quoted as published on 2026-08-31.
Related
More from Signals
Open the tool.
Ten production categories, three minutes, and a ranked list of what to fix first.
Check your own stack