Cafiyn Pulse
← Startup Signals
5 min readUpdated 2026-08-31Issue 1

Your model comparison is measuring the wrong number.

Every price list is per token. Your bill is per finished piece of work. Those two numbers can rank models in opposite orders.

ByKarthik KumarCafiyn Innovations
At a glance
What providers publish
Cost per million input and output tokens
What you actually pay for
Tokens across every attempt until the task succeeds
Hidden multiplier
Retries, reasoning tokens, and replayed context
Worked example
Anthropic reports Opus 5 beating Opus 4.8 at lower cost per task
Practical rule
A pricier model that succeeds first time can be the cheaper one

Two numbers that disagree

Provider pricing pages are all denominated the same way: so much per million input tokens, so much per million output tokens. It is a fair unit for the provider, because tokens are what they process. It is a poor unit for you, because tokens are not what you are buying. You are buying completed work.

The gap between those shows up the moment a task takes more than one attempt. If a cheaper model needs two passes and a validation retry where a more expensive one succeeds immediately, the cheaper model has just cost you three calls plus the engineering time to handle the failure path. The per-token price list ranked them one way. Your invoice ranks them the other.

Where the extra tokens hide

Three things inflate real cost above the sticker price, and none of them appear on a pricing page.

Retries are the obvious one. Any task with a validation step, structured output that must parse, code that must compile, an answer that must cite a real source, has a failure rate, and every failure is a call you paid for and threw away.

Reasoning tokens are the second. A model that thinks before answering generates tokens you never see but are billed for as output. A short visible answer can carry a much larger output bill than its length suggests.

Replayed context is the third, and it is the one that grows. Every turn where you resend conversation history is history you pay for again. A conversation at turn fifty costs far more per turn than the same conversation at turn five, even though nothing about the model changed.

What the current generation is claiming

This is why the framing in recent model releases has shifted. Anthropic describes Claude Opus 5 as more than doubling Opus 4.8's performance on Frontier-Bench v0.1 at a lower cost per task, and as landing within 0.5% of the top CursorBench 3.2 score at roughly half the cost per task.

Read carefully, those are not per-token claims. They are claims about the total cost of getting a job finished, which is the honest way to compare and also the harder thing to verify. Treat a vendor's cost-per-task figure the way you would any vendor benchmark: as a hypothesis to test on your own workload, not a settled result.

Working out your own number

The measurement is not complicated. Pick one task your product performs often. Run it a hundred times against each candidate model. Record total tokens spent including every failed attempt, and record the success rate against whatever "correct" means for that task.

Divide total cost by successful completions. That is your cost per task, and it is the only number that predicts your bill. It is common for the ranking it produces to differ from the ranking on the pricing page, which is precisely why it is worth an afternoon.

What to do

  1. 1Pick your highest-volume task and define what a successful completion means, precisely enough to measure automatically.
  2. 2Run a hundred trials per candidate model, counting tokens from every attempt including failures.
  3. 3Divide total spend by successful completions. Compare that number, not the pricing page.
  4. 4Instrument retry rate in production. A rising retry rate is a cost increase that no price change will explain.
  5. 5Re-run the comparison when you change the prompt. Prompt changes move success rate, and success rate moves cost per task more than token price does.

Common questions

What is cost per task?

Total spend divided by successfully completed jobs, counting the tokens burned on failed attempts. It captures retries, reasoning tokens, and replayed context, none of which appear on a per-token price list.

Can a more expensive model be cheaper?

Yes, and it is common. A model with a higher per-token price that succeeds on the first attempt can cost less per finished task than a cheaper model that needs two or three tries.

Why do reasoning models cost more than their output length suggests?

They generate internal reasoning tokens before the visible answer, and those are billed as output. A brief response can carry a substantial output charge.

How many trials do I need?

A hundred per model is usually enough to separate candidates on a task with a meaningful failure rate. If the models are close, the difference probably will not decide your bill anyway.

Sources

Read against the primary documentation rather than secondary coverage. Where a figure comes from a provider's own docs, it is quoted as published on 2026-08-31.

Related

Model your bill across 22 providers →Per-token pricing for every flagship model →What a code assistant costs per developer →

More from Signals

OpenAI removes o1, o3-mini, o4-mini and GPT-4 Turbo on 23 October. The replacements it names run from 5.5 times the price to 97 percent less.
Eleven models and six fine tuned models go dark in four days. Here is OpenAI's own mapping, what each landing costs on a typical month of traffic, and the places where the named replacement is not the cheapest or cannot be used at all.
OpenCode says a breach hit 1.4 percent of Console users who linked an AI provider. Rotate those keys before the root cause is published.
OpenCode has confirmed the breach and has not said how it happened or what was taken. Here is what the vendor said, who is exposed, the order to work in, and how to limit the damage next time.
GitHub Actions workflow lockfiles are in public preview. Here is how to turn them on, check them in CI, and what they do not protect.
A tag can move after you reviewed it. A lockfile makes the runner use the commit you reviewed. Here is the setup, the CI check, the policy, and what a lockfile cannot tell you.
Next.js 16.4.1 and 15.5.28 fix two critical remote code execution flaws. For one of them there is no workaround.
Two of the three fixes were postponed from September. Here is what each flaw needs in order to be reachable, the one workaround that exists, and what the announcement leaves out.
A poisoned npm release that punishes you for revoking the stolen token. The order of your response is the whole defence.
Most incident guidance starts with revoking the leaked token. Here that is the trigger. What StepSecurity found, the order it recommends, and why a valid provenance attestation did not help.
Claude Haiku 5.5 costs a tenth of Haiku 4.5 per token. One token past 100,000 it costs half.
The rate card is a tenth of Haiku 4.5. The footnote is a threshold that moves the price fivefold and that a tokenizer change pushes you toward.
Gemini 4 Argon launched at $2 and $10 and goes to $4 and $20 on 1 December. Price your product on the second number.
Every tier on the pricing page doubles on the same date. What changes, what the number includes, and what to do if you are building on the launch rate.
Claude Haiku 5.5 is not a model id swap. Five things that worked on Haiku 4.5 now return a 400.
The model id is the easy part. The requests that now fail, the response parsing that changes, and the details that fail quietly.
Next.js shipped 16.3.7 and 16.3.8 a day apart. Only one of them is the security release.
Two patch releases a day apart, one of them a decoy if you are not reading carefully. And the 15.5 line does not get the high severity fix at all.
The EU AI output marking rule is not a December deadline. For most teams it started in August.
Almost every summary of this gives one date. There are two, and which one applies to you depends on when you shipped, not on how big you are.
Sonnet 5.5 cut the rate by a third and the token count went up by about 30 percent.
A third off the per-token rate is not a third off the bill when the same text now costs more tokens. Here is the arithmetic, and the five ways your code can break.
Your AI gateway holds every provider key you have. A 9.9 just landed in LiteLLM.
An AI gateway concentrates every upstream key into one process. That is the point of it, and it is why a privilege escalation there is worse than it looks.
The table your coding agent created does not have row level security on it.
Dashboard-created tables get row level security by default now. API-created tables do not, and API is how your coding agent works.
A 9.5 in next/og. Whether it reaches you depends on one line in your OG route.
The advisory rates it 9.5, but it only fires if your OG image route puts attacker-controlled text into SVG. Here is how to find out which side of that line you are on.
OpenAI and Anthropic both cut prices on 22 September. One of them costs you a refactor.
Two frontier price cuts in one day is a good week for anyone paying a token bill. The migration cost is not evenly distributed, and one of the two prices comes with an expiry hedge.
A package install reads your home directory. This week two of them did it on purpose.
The implant did nothing clever. It read the files your shell reads, in the place your credentials live, and sent them somewhere else. That is the whole attack, and it works.
GitHub is switching off SHA-1 SSH keys. Find out now whether yours is one of them.
The keys at risk are the old ones, on the machines nobody logs into, in the pipelines that have run untouched for three years. Those are also the ones that will fail loudest.
Every third-party script tag is a standing grant to run code on your visitors.
A marketing form, a chat widget, an analytics snippet. Each one is permission for someone else's server to execute whatever it sends, on every page it sits on, forever.
1.8 million Android apps were scanned for hardcoded keys. Here is how to check yours.
Finding secrets in shipped apps is now a cheap, automated pipeline. Anything compiled into an APK, an IPA or a JavaScript bundle should be treated as already published.
OpenAI's Agents API will hold your agent state. What breaks if you ever need to leave.
Two major providers moved into the agent state layer in the same week. A managed harness is a real shortcut, as long as you keep a copy of what it holds.
Meta's Muse agent can now check out at your store. What merchants should change.
Agent purchases are arriving through wallets and single-use cards, not through merchants adopting a new protocol. Many merchants will receive them without choosing to.
What Brevo's SSO incident teaches about multi-tenant SAML and hijacked sending accounts.
One incident, two lessons: why email authentication did not help, and the tenant-scoping mistake anyone shipping SSO can make.
Cloudflare now sorts bots into Search, Training and Agent. What it means for your site and your agent.
The same change lands on two audiences. Site owners can now accidentally block search; agent builders will find more doors closed.
OpenAI has announced ten dated deadlines between 23 October 2026 and 1 April 2027. Here is the calendar.
Ten dated deadlines in six months, every one announced in public. The failure mode is never the notice. Updated 19 October with the entries added since September.
Nvidia bought Hugging Face. Your open-weight fallback is now a vendor relationship.
The commitments say nothing breaks. The structural change is that "we can always self-host" is now a dependency on one vendor rather than on a neutral commons.
The Videos API goes away on 24 September and OpenAI names nothing to move to.
Every other deprecation on OpenAI's page offers a migration target. This one does not, which makes it a different kind of problem.
Whisper shuts down on 26 February. The risk is not the deadline.
Five months is a generous window. Whisper is also typically wired into a background job that fails into a queue rather than into a user's face.
The Assistants API is gone. Your app may not have told you yet.
It shut down on 26 August 2026. The replacement is not a drop-in, and the most common migration mistake changes your model's behaviour without throwing an error.
One Copilot code review costs 13 requests. Here is the rest of the maths.
Copilot now bills on usage rather than seats. The headline change is simple. The multipliers underneath it are where teams get surprised.
Row-level security is on by default now. That is not the same as being protected.
Lovable fixed the default. The gap that actually leaks data was never the default, it was the policy, and no platform can write that for you.
An agent with production log access is a new identity. Treat it like one.
Always-on agents that read production logs are arriving fast. The access-control question they raise is old and well understood, which is the good news.

Open the tool.

Ten production categories, three minutes, and a ranked list of what to fix first.

Check your own stack