Learn · Beginner

Inference cost and token economics: why output tokens cost more than input

Model providers bill separately for the tokens you send and the tokens the model generates, and the generated ones typically cost three to five times more. That is not a pricing preference -- it reflects a hard asymmetry in how transformers run. Reading your prompt is one parallel pass through the network; writing a reply is one full pass per word. Understanding that asymmetry is what separates people who can forecast an AI product's unit economics from people who are surprised by the invoice.

A concrete example: when OpenAI reduced GPT-5.6 Sol in August 2026, input fell from $5 to $4 per million tokens while output fell from $30 to $20. The headline said "over 20 percent." For anyone running an agent loop, the real number was 33 percent, because agent loops are almost entirely output.

Where the asymmetry comes from

A transformer processes a prompt in what is called the prefill phase. Every token in your input can be looked at simultaneously, because they are all already known -- the matrix multiplications run across the whole sequence at once, saturating the hardware. This is the arrangement described in Attention Is All You Need by Ashish Vaswani and colleagues at Google, and parallelism over the sequence is the specific property that made transformers replace recurrent networks.

Generation cannot work that way. The model does not know its second word until it has produced the first. So decoding is strictly sequential: one full forward pass through every layer, to produce one token, then repeat. A thousand-token answer is a thousand passes. A thousand-token prompt is one.

Worse, those passes are inefficient. During generation you are pushing a single token through weights that occupy tens of gigabytes, so the accelerator spends most of its time moving parameters from memory rather than computing with them. This is the memory-bandwidth wall covered in why LLM inference is memory-bound, and it was analyzed carefully by Reiner Pope and colleagues in Efficiently Scaling Transformer Inference.

The analogy that fits: reading a page is one glance across it; writing a page is one word at a time, and after each word you reread everything you have written so far. Both involve the same page. Only one of them is cheap.

The state that grows

Rereading is not a metaphor. Every generated token attends to all preceding tokens, and recomputing that from scratch each step would be absurd, so servers keep a cache of the attention state -- the KV cache. It grows linearly with conversation length and it lives in the same scarce memory as the weights.

This has two economic consequences. First, long conversations get progressively more expensive per token, because the state being carried is larger. Second, cache memory limits how many users a server can handle at once, which sets the provider's cost per request. Woosuk Kwon and colleagues attacked exactly this in PagedAttention, the technique behind the vLLM serving engine, which manages cache memory in pages like an operating system and thereby raises how many conversations fit on one machine. Serving efficiency is not a back-office concern; it is upstream of the price you pay.

What actually moves your bill

Cache the repeated part. Most production prompts are mostly constant -- the same system instructions, the same retrieved documents, the same few-shot examples, resent verbatim on every call. Providers let you mark that prefix as cacheable and charge a fraction of the normal rate for a hit. See prompt caching. For a chatbot with a long system prompt, this is frequently the single biggest saving available, and it requires no model change.

Send cheap work to cheap models. Classification, routing, extraction, and formatting rarely need a frontier model. Running a small model first and escalating only when needed is the pattern in model routing and cascades, and it is exactly what agent harnesses are doing internally when they request different capability tiers for different sub-jobs.

Watch reasoning tokens. Reasoning models generate long internal chains before answering, and you pay for that generation even when it is hidden from you. A model that thinks for two thousand tokens to produce a fifty-token answer bills you for two thousand and fifty output tokens. Test-time compute is genuinely powerful and it is not free -- it converts money into accuracy, which is a fine trade only if you meant to make it.

Consider running it yourself. Quantization has pushed capable models onto single consumer cards, which turns a per-token bill into a fixed hardware cost plus electricity. That flips the arithmetic entirely at high volume and rarely makes sense at low volume.

Know the research levers. Speculative decoding, introduced by Yaniv Leviathan and colleagues at Google in Fast Inference from Transformers via Speculative Decoding, uses a small draft model to guess several tokens ahead and a large model to verify them in one pass -- amortizing the expensive sequential step. See speculative decoding. You do not implement this yourself; you benefit from it when your provider does, and it is part of why prices keep falling.

The takeaway

Before building anything on a model API, estimate the ratio of tokens read to tokens written for your actual workload, then price it against the output rate rather than the input rate. Summarization is input-heavy and cheap. Code generation, long-form writing, and multi-step agents are output-heavy and are where budgets go. The single most useful habit is to instrument token counts per request from day one -- almost every cost surprise in production is a workload whose output volume nobody measured until the bill arrived.

Ground Truth. (2026, August 24). Inference cost and token economics: why output tokens cost more than input. Ground Truth. https://groundtruth.day/learn/inference-cost-and-token-economics.html

@misc{groundtruth:inference-cost-and-token-economics,
  title  = {Inference cost and token economics: why output tokens cost more than input},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/inference-cost-and-token-economics.html}
}

Topics: fundamentals · inference · cost · serving · kv-cache · economics