How to think about LLM inference cost

One user request can trigger many billed calls; count each once before diagnosing repeated context, retries or expensive tails.

Problem

Three model calls consume 1,200, 1,400 and 1,600 input tokens, plus 200 output tokens each. Each input includes the same 1,000-token prefix. The first use is uncached; the next two receive a prefix discount. Tools cost 0.001 dollars in total.

Use fictional prices per million tokens: 1 dollar uncached input, 0.2 cached input and 4 output. There are no additional cache-write charges in this example. Compute the request total and compare it with the same calls without caching. These are teaching prices, not current provider quotes.

Picture: Price Three Rows, Not Five Overlapping Buckets

The prefix and history are already inside each input count. Do not add them again as separate token charges. Add tools once: the full request costs 0.006 dollars. Without caching, it costs 0.0076 dollars; the saving is about 21.05%. The 80% cached-token discount does not apply to the entire request.

A Practical Cost Model

For each call, input is partitioned into uncached and cached tokens. Rates below have currency-per-token units; add explicit write/tool charges and allocated costs only once.

from decimal import Decimal


def model_cost(calls):
	total = Decimal("0")
	million = Decimal("1000000")
	for call in calls:
		input_tokens = call["input_tokens"]
		cached = call["cached_input_tokens"]
		output = call["output_tokens"]
		if not 0 <= cached <= input_tokens or output < 0:
			raise ValueError("invalid token partition")
		total += (
			Decimal(input_tokens - cached) * Decimal("1")
			+ Decimal(cached) * Decimal("0.2")
			+ Decimal(output) * Decimal("4")
		) / million
		total += call.get("cache_write_cost", Decimal("0"))
	return total


calls = [
	{"input_tokens": 1200, "cached_input_tokens": 0, "output_tokens": 200},
	{"input_tokens": 1400, "cached_input_tokens": 1000, "output_tokens": 200},
	{"input_tokens": 1600, "cached_input_tokens": 1000, "output_tokens": 200},
]
assert model_cost(calls) == Decimal("0.005")
assert model_cost(calls) + Decimal("0.001") == Decimal("0.006")

Reconcile request/call/attempt IDs, usage categories, cache state, rate version and tool receipts against billed totals. Include every retry, cancellation and failed task that incurred charges.

Change 1: Stable Prefixes Dominate

New task. Most input repeats an authorized prefix.

Picture and change. Measure eligible cache hits, not just repeated text. Check provider/model support, exact-prefix rules, expiry, write/miss charges and tenant isolation. Compare safe prompt trimming under the same evaluation.

Check. Cached input does not discount outputs, tools or newly added history. Authorization and source freshness still apply independently of cache reuse.

Use when. Measured eligible repeated work makes caching worthwhile.

Change 2: Long Agent Runs Dominate

New task. Retries and growing tool history make some tasks expensive.

Picture and change. Bound steps, retries, tokens, time and money at enforced limits. Summarize only while preserving source evidence, authority, unresolved effects and stop conditions. Charge the summary calls too.

Check. With first input $I_0$ and $h$ additional history tokens per step, cumulative input is

Our three-call ledger fits $I_0=1200$, $h=200$, giving 4,200 input tokens. Bounded fixed-size context instead gives approximately linear cumulative input, excluding additional compaction work.

Use when. Measured multi-call tails dominate spend. An exhausted run is incomplete, not a cheap success.

Preserve Release Requirements

Compare quality, severe failures, authority, latency and cost per successful task on the same population. Keep a rollback for prompt, routing, retrieval, model and budget changes. Lower cost can come from removing waste or from losing useful work; distinguish them.

Failure: Charge the Same Work Twice

Prompt and retrieval tokens cannot be added again after counting full input. Actual tail calls already appear in the ledger; do not add another arbitrary tail charge. A dimensionless multiplier cannot be added directly to currency.

Transfer Check

Must cumulative input grow quadratically with the number of calls?

Reasoned Answer

No. Quadratic growth requires each call to resend roughly linearly growing history. Fixed-size context can grow approximately linearly, and cache pricing changes cost separately from token count.