Training a model is a one-time headline number. Inference is the recurring bill that scales with every user, every request, and every reasoning token, and it quietly decides whether an AI product survives. With an interactive calculator for your own numbers.
A deep read: the full picture, with the receipts.
The eye-watering numbers in AI coverage are almost always about training: the cost to build a model from scratch. Those numbers are real, but they are a one-time, mostly fixed cost. The number that actually decides whether an AI product lives or dies is inference, the cost of running the model every single time someone uses it. Training is the down payment. Inference is the rent, and the rent never stops.
Training is a sunk cost; inference is a tax on growth#
Once a model is trained, that money is spent. Inference is different in a way that should keep founders up at night: it scales directly with usage. Every active user, every request, every retry adds to the bill. The more successful your product, the more it costs to run, and that cost arrives in real time whether or not the revenue does.
This inverts the usual software intuition. Traditional software gets cheaper per user as you grow, because the cost of serving one more user rounds to zero. AI products do not get that gift for free. More users means more inference means more cost, unless you engineer your way out of it. A viral hit can become an existential threat: the spike in usage you celebrated on Monday is the bill you cannot pay on Friday.
Where the compute money goes over a deployed model's life
The exact split varies by model and how heavily it is used; the direction is the reliable part. For anything with real adoption, the recurring cost of serving the model dwarfs the one-time cost of building it.
How the inference meter actually runs#
To design around inference cost, it helps to know exactly what you are being charged for. Models bill by the token, on both sides of the exchange: the tokens you send in (the system prompt, the user's message, chat history, and any retrieved documents you stuff into the context) and the tokens the model generates back. Output tokens are almost always priced higher than input tokens, often several times higher, because generating text one token at a time is the expensive part.
Underneath the per-token price, you are really renting time on scarce accelerators. The dominant cost is keeping those chips busy, which is why providers batch many users' requests together and smooth it all into a per-token rate. That leaves you two levers that move your bill more than any other: how many tokens ride along in every request, and how many the model produces in reply. Long contexts and long answers are the quiet budget-killers, and both are easy to let creep upward without noticing.
At 1,000,000 requests/mo, Route + cache repeats is cheaper.
Illustrative, based on typical 2026 token prices for a roughly 2,000-token request (a frontier tier near $5/$25 per million input/output tokens, a small tier near $1/$5). Every option here is a real s
Reasoning models moved the goalposts#
The 2026 wrinkle is reasoning models. OpenAI's reasoning line, Claude's extended thinking, Gemini's thinking modes, and DeepSeek-style reasoners all generate a separate block of thinking tokens before the visible answer, and you pay for every one of them regardless of whether the product shows that thinking to you or hides it. A reply with a few hundred visible tokens can burn an order of magnitude more in reasoning, and because that thinking is generated text, it lands in the higher-priced output bucket. Point a reasoning model at every request by default and you can end up paying far more for the model's deliberation than the answer is worth.
Used deliberately, on genuinely hard problems and with a cap on how long the model is allowed to think, reasoning models earn their cost. Used as the default setting for everything, they quietly multiply the bill on work that a fast, non-reasoning model would have finished for a fraction of the tokens.
The trap of the free-and-generous launch#
The failure pattern is easy to spot in hindsight. A product launches with a generous free tier and an expensive model behind every interaction. Users love it. Usage climbs. And the cost of serving that usage climbs right alongside, with no revenue keeping pace. The common ways this goes wrong compound on each other:
Growth, in that setup, makes the problem worse, not better. The product becomes a machine for converting attention into losses, and the only fixes left are painful: degrade the free experience, raise prices, or cap the very usage you spent so much to attract.
Where the savings actually come from#
Teams that survive treat inference cost as a first-class engineering problem, not an afterthought. The biggest levers are unglamorous, and none of them are exciting. All of them are the difference between a product with a future and a demo that bleeds out.
What this signals#
The industry's obsession with training costs has been a distraction from the number that matters at the product level. Training costs make headlines because they are huge and concentrated. Inference costs decide outcomes because they are constant and tied directly to success, and reasoning models and always-on agents are pushing them up, not down. As AI moves from demos to deployed products at scale, the winners will be the teams that treat inference as their core operating cost and design around it from day one, the same way any business watches the cost of the goods it sells. The question is no longer can you build it. It is can you afford to run it once people actually use it.
What is the difference between AI training cost and inference cost?
Training is a one-time, mostly fixed cost to build a model from scratch. Inference is the recurring cost of running the model every time someone uses it, and it scales directly with usage. Over a model's deployed life, inference typically accounts for the large majority of the compute spent on it.
Why does inference cost make AI products harder to scale than traditional software?
Traditional software gets cheaper per user as you grow because serving one more user rounds to zero. With AI, more users means more inference and more cost, so growth can make the problem worse rather than better unless you engineer around it.
What are you actually paying for when you pay for inference?
You pay by the token, for both the tokens you send in (prompt, context, chat history, retrieved documents) and the tokens the model generates back. Output tokens are usually priced higher than input tokens. Underneath, you are renting time on scarce accelerators, so the two things that move your bill most are how long your inputs are and how much the model produces.
Why do reasoning models cost more to run?
Reasoning models generate a separate block of thinking tokens before the visible answer, and you pay for all of them at the higher output-token rate, whether or not the product shows that reasoning to you. A short answer can consume an order of magnitude more tokens in reasoning, so using a reasoning model by default, rather than only for genuinely hard tasks, can multiply your inference bill.
What causes AI inference costs to spiral out of control?
Common causes include calling a frontier model for tasks a cheap model could handle, leaving reasoning on for every request, no caching of repeated requests, verbose prompts and bloated context, retries and agent loops that multiply calls, and a free tier too generous to ever convert fast enough to cover itself.
How can teams reduce AI inference costs?
Route easy requests to small cheap models and reserve frontier models for the hard tail, cache aggressively to take advantage of discounted repeated tokens, trim prompts and cap output length, reserve reasoning for hard tasks with a bounded thinking budget, and set quotas so a few heavy users cannot sink the unit economics.
Why is a generous free tier with an expensive model risky?
Usage climbs with the free tier while no revenue keeps pace, so serving costs rise with adoption. The remaining fixes are painful: degrade the free experience, raise prices, or cap the usage you spent to attract.
Sources
- Kaplan et al., Scaling Laws for Neural Language Models (arXiv)arxiv.org
- Hoffmann et al., Training Compute-Optimal Large Language Models / Chinchilla (arXiv)arxiv.org
- Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention / vLLM (arXiv)arxiv.org
- Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (arXiv)arxiv.org
Discussion