Tracking LLM Costs in Production: What to Measure and Why

An AI feature that works in a demo can quietly become your biggest bill. Here is what to instrument before that happens.

Hafeez Baig•Dec 16, 2025•10 min read

Introduction

Most infrastructure cost is bounded by something physical. A server has a fixed size and a fixed price. A database has a disk that fills up. When you go over, something breaks loudly and you find out.

LLM cost does not work that way. You pay per token, a chunk of text roughly the size of a short word or part of one. Every call you make, and every word the model writes back, adds to the bill. Nothing imposes a ceiling. If traffic doubles, the bill doubles. If someone calls your endpoint in a loop, the bill does whatever they want it to.

A demo costs almost nothing, so nobody instruments it. Then it ships, and the first surprise arrives as an invoice rather than an alert. This post covers what to measure, and what to limit, before that happens.

1. Why Token Cost Is Unbounded by Default

Two properties make LLM spend behave differently from the rest of your stack.

It is usage based. You are not renting capacity, you are buying each unit of work. There is no idle cost and no natural cap. The provider is happy to keep serving you.

The cost of a single call is variable. A traditional endpoint costs about the same every time it runs. An LLM call can cost a fraction of a cent for a short question and far more for a long document, because the price scales with how much text goes in and how much comes out.

That second point is what teams underestimate. A request count tells you nothing until you know how big those requests were.

2. The Unit Economics Question

Before optimising anything, answer one question: what does one user action cost you?

Not what the month cost. One action. One summary generated, one chat message answered, one document processed.

With that number, other questions become answerable. How many actions does a paid plan cover before that customer stops being profitable? Which feature is worth optimising, and which is noise?

Work it in symbols rather than guesses. Suppose one action costs C and you serve N of them per month. Your bill is roughly C times N. That is obvious written down, but most teams cannot fill in either variable, because nothing records C and their analytics count page views rather than model calls.

If you build one thing from this post, build the thing that tells you C.

3. What to Log on Every Call

Cost is arithmetic over data you already have at the moment of the call. The provider returns token counts in the response. Write them down.

Field	Why you need it
`model`	Prices differ per model. Without this, token counts are meaningless.
`input_tokens`	Everything you sent: system prompt, context, user text.
`output_tokens`	Everything the model wrote back. Usually the pricier half.
`latency_ms`	Cost and speed trade against each other. Track both or you optimise blindly.
`feature`	Which product surface made the call. The most useful field here.
`user_id` or `tenant_id`	Who the spend belongs to.
`cache_hit`	Whether you paid full price or a reduced one.
`retry_count`	A retried call is a call you paid for twice.

Retries are the classic invisible cost. Your error rate looks fine because the retry succeeded, and the failed attempt still consumed tokens. If your client library retries automatically, find out and record it.

4. A Logging Wrapper

Never call the provider SDK directly from feature code. Put one thin wrapper in front of it and route everything through that.

type CallMeta = {
  feature: string;      // e.g. "ticket_summary"
  userId: string;
  cacheHit?: boolean;
};

// Prices are configuration, not constants in code.
const RATES = getRateTable(); // { [model]: { inPer1k, outPer1k } }

async function callModel(
  model: string,
  prompt: string,
  meta: CallMeta,
  opts: { maxOutputTokens: number },
) {
  const startedAt = Date.now();
  let attempts = 0;

  const res = await withRetry(() => {
    attempts += 1;
    return provider.messages.create({
      model,
      max_tokens: opts.maxOutputTokens, // always set this
      messages: [{ role: "user", content: prompt }],
    });
  });

  const inTok = res.usage.input_tokens;
  const outTok = res.usage.output_tokens;
  const rate = RATES[model];
  const cost = (inTok / 1000) * rate.inPer1k + (outTok / 1000) * rate.outPer1k;

  metrics.record("llm.call", {
    model,
    feature: meta.feature,
    userId: meta.userId,
    inputTokens: inTok,
    outputTokens: outTok,
    estimatedCost: cost,
    latencyMs: Date.now() - startedAt,
    cacheHit: meta.cacheHit ?? false,
    retryCount: attempts - 1,
  });

  return res;
}

Note three things. feature and userId are required arguments, so nobody adds an untracked call by accident. max_tokens is always set. And the price table is configuration, because provider prices change and a cost report should not need a deploy to stay correct.

Reconcile estimatedCost against the real invoice occasionally. If the two drift apart, something is calling the provider outside your wrapper.

5. How Token Counts Drive Cost

Let Ci be the price per thousand input tokens and Co the price per thousand output tokens. Cost per call is:

(input tokens / 1000) x Ci + (output tokens / 1000) x Co

A few call shapes show where the money goes. The numbers are illustrative, not measured:

Call shape	Input tokens	Output tokens	Cost per call
Short classification	200	20	0.2 Ci + 0.02 Co
Chat turn with history	2,000	300	2 Ci + 0.3 Co
Document summary	20,000	500	20 Ci + 0.5 Co
Long generation	500	4,000	0.5 Ci + 4 Co

The summary is dominated by input, so trimming context beats trimming the answer. The long generation is dominated by output, where a token cap is the control that works. Output is usually priced higher than input, so a small output count can still outweigh a large input one. Which lever helps depends on the shape of the call, which is why you log both numbers separately.

6. Attribute Cost, Do Not Just Total It

A monthly total tells you that you have a problem. It does not tell you where.

Attribute to a feature. Grouped by feature, the picture is almost always lopsided. One surface dominates, usually one nobody thought of as expensive, because it quietly re-sends a large context on every keystroke or page load. That is invisible in a total and obvious in a breakdown.

Attribute to a user or tenant. A small number of accounts tend to generate a large share of calls. You need to know whether your heaviest user is a valuable customer or someone abusing a free tier. Without a per user figure, the only lever you have is switching the feature off.

7. The Levers, in Order of Impact

When cost needs to come down, work in this order. Most teams start too far down the list.

Do not call the model at all. The cheapest call is the one you never make. A surprising share of LLM calls answer questions a lookup, a regular expression, or a database query could answer exactly and instantly.
Cache. Identical or near identical requests are common in real traffic. Cache the finished result, keyed on the input. A hit costs nothing and returns immediately.
Use a smaller model. Classification, extraction, and short summaries are often handled well by a cheaper model. Test on your actual data rather than assuming.
Shorten the prompt. Long system prompts, whole documents pasted in when a section would do, and conversation histories replayed every turn are all avoidable.
Cap output tokens. An unbounded response is an unbounded charge.
Batch. If work is not time sensitive, group it. Some providers price asynchronous processing lower.

The first item is worth sitting with. The interesting cost wins are architectural, not parameter tweaks.

8. Prompt Caching

Many applications send a large, identical block of text at the start of every request: a long system prompt, a set of instructions, a document the user is asking about. It is re-sent and re-charged on every call even though it never changes.

Prompt caching lets the provider hold that stable prefix, so repeat calls sharing it cost less than sending it fresh. The mechanics and the discount vary by provider, so check the current documentation for the one you use rather than trusting a blog post, including this one.

The design principle transfers regardless: put the stable content first and the variable content last. A timestamp or a user name near the top of an otherwise static prompt can defeat the whole thing.

9. Unbounded Input Is a Cost Bug

Treat this pattern as a defect on sight:

// Do not do this.
const summary = await callModel(model, `Summarise this: ${req.body.text}`, meta, opts);

Whatever the user sends becomes what you pay for. If someone pastes a book, you buy a book. If someone sends a hundred thousand tokens of nonsense, you pay full price for the privilege of receiving nonsense.

The fix is a limit, applied in your wrapper so it covers every path:

Cap input length before the call. Measure it, then reject or truncate anything over your threshold.
Count tokens, not characters, where precision matters. Characters are a fine rough proxy for a first pass.
Decide what happens at the limit. Silent truncation produces bad output and confused users. A clear rejection is usually better. Chunking is sometimes right, but it multiplies calls, trading one cost for another.

This covers anything you did not write yourself: uploaded files, scraped pages, retrieved documents, tool results, third party responses. All of it is user supplied input as far as your bill is concerned.

10. Rate Limits, Quotas, and Hard Caps

Instrumentation tells you what happened. Limits decide what is allowed to happen.

Per user rate limits. A cap on calls per minute for one account. This stops a runaway client loop, an over eager retry, or one user monopolising capacity. It is the cheapest control to add and the one most often missing.

Per user or per tenant quotas. A budget over a day or a month. This is where product decisions belong: a free tier gets a small allowance, paid tiers get more, and reaching the quota shows the user a clear message rather than showing you a clear invoice.

Hard budget caps at the account level. Most providers let you set a spending limit and alert thresholds. Set the alert well below the cap, so you find out while there is still time to act. A cap with no alert means your first signal is your feature going down.

Alert on the rate of change, not just the level. A bill reaching its threshold on the last day of the month is normal. The same threshold on day three is an incident.

11. The Open Endpoint

The most expensive mistake here is also the simplest, and it usually ships without anyone noticing.

You add an endpoint that takes text from the request body, passes it to a model, and returns the answer. It works. You are testing, so authentication comes later. It ships without authentication.

You have now published a free, anonymous proxy to a paid API, funded by your key. Endpoints like this get found, not by someone reading your code but by automated scanning that probes for exactly this shape. The traffic is a script, and it does not stop at a reasonable hour.

The rule I would apply without exception: never accept arbitrary content from a request body and forward it to a model. Any model call reachable from the internet needs:

Authentication, so every call is attributable to an identity, including on a feature you intend to make public later.
A per identity rate limit, enforced server side.
A validated, bounded input. Check the type, the length, and the shape.
A structure you control. Your code owns the prompt. The user's text is a bounded value inside it, never the whole thing.
A capped output. max_tokens set on every path.
Server side key handling. If a key can reach the browser, it will.

That fourth point is a security concern as well as a cost one. If a user supplies the entire prompt, they can also supply instructions. That is the entry point for prompt injection, where a user's text talks past your intent and gets treated as a command. Bounding the input and owning the surrounding structure narrows both problems at once.

Conclusion

LLM cost is not hard to control. It is hard to control retroactively, once a feature is live, spending is growing, and nobody can say which surface or which user is responsible.

The work is unglamorous and mostly front loaded:

One wrapper, so every call is logged with model, tokens, latency, feature, user, cache status, and retries.
One number you can quote on demand: what a single user action costs.
Limits on input size, output size, call rate, and total spend.
Authentication on anything the internet can reach.

That is a day of work at the start of a project and a much worse week at the end of one. If your AI feature is live and you cannot say what one call costs, start there. Everything else follows from that number.

Written for engineers putting AI features into production, where the bill is part of the design.