A model call costs cents. A bad retry loop costs a surprising amount of money in an afternoon. The first AI bill that surprised us was traced to a single feature where, on a particular error path, the code retried up to thirty times before giving up. The function "worked." The bill did not.
This essay is about the small accounting library we wrote in response, which has paid for itself many times over.
What the library actually is#
A wrapper around every model call that:
- Counts tokens before sending.
- Records actual usage from the response.
- Attributes both to a (tenant, feature) pair.
- Refuses to dispatch the call if the tenant is over budget.
About 200 lines of code total. The whole thing is a thin layer between "my code" and "the model SDK." It looks like this:
type Caller = { tenantId: string; feature: string };
interface ModelCall {
prompt: string;
model: 'cheap' | 'medium' | 'expensive';
maxOutputTokens: number;
}
class GuardedClient {
async call(caller: Caller, req: ModelCall): Promise<string> {
const inputTokens = await this.countTokens(req.prompt, req.model);
const projectedCost = this.priceOf(req.model, inputTokens, req.maxOutputTokens);
if (!await this.budget.canAfford(caller, projectedCost)) {
throw new BudgetExceeded(caller, projectedCost);
}
const start = Date.now();
const result = await this.upstream.complete(req);
const elapsed = Date.now() - start;
await this.budget.charge(caller, {
inputTokens,
outputTokens: result.usage.output_tokens,
cost: this.priceOf(req.model, inputTokens, result.usage.output_tokens),
elapsed,
feature: caller.feature,
});
return result.text;
}
}Five things are loadbearing in those few lines.
1. Token-count before send#
The cost of a call is roughly input_tokens × input_price + output_tokens × output_price. We can know input_tokens exactly before sending; we
have to guess output_tokens, which is why maxOutputTokens is a
required parameter. The projection is "input + max output;" the actual
charge is "input + actual output."
2. The budget check is hard, not soft#
canAfford returns false if the tenant's daily spend plus the
projected cost would exceed their cap. We throw, not warn. Soft
warnings get ignored.
3. The (tenant, feature) attribution#
Every charge is attributed to both. This lets us answer two different questions:
- "Which tenants are most expensive?" (sum by tenant)
- "Which features are most expensive?" (sum across tenants by feature)
Cost-per-feature is the more interesting one. A typical finding is that one barely-used feature is responsible for a large share of spend, because it runs a verbose model on a huge prompt and the average user clicks it twice a session. Rewriting it with a smaller prompt and a cheaper model cuts cost-per-call sharply.
4. Wall-time, not just tokens#
We log elapsed time alongside cost. Slow calls are not always expensive, but a feature that always takes 9 seconds is a UX problem even if it costs nothing.
5. Per-tenant, not per-user#
The budget is on the tenant (the company), not the individual user. This matches how costs are billed. It also avoids the bug where an admin user with privileges burns the user-budget of a colleague who did not consent.
What we set the budget to#
Three budget tiers, declared per tenant plan:
| Plan | Daily budget | Monthly budget | Action when exceeded |
|---|---|---|---|
| Free | small | small | Block, suggest upgrade |
| Pro | larger | larger | Block, alert tenant admin |
| Enterprise | per-contract | per-contract | Soft cap, alert support |
Two unintuitive choices:
Free tier has a non-zero budget#
We considered "free tier gets nothing." It is hostile. A small daily budget covers a handful of cheap-model calls, which is enough for a user to evaluate the feature without us subsidising thousands of calls per free user.
Enterprise has a soft cap#
The hard cap on Pro is the right default; for Enterprise, blocking their workflow because their admin forgot to update the cap on contract renewal is bad business. Soft cap, alert, conversation.
What we instrument#
A small dashboard with five panels:
- Spend per hour, last 24 hours. Tells us when a spike is happening.
- Top 10 features by today's spend. Tells us where the spike is.
- Top 10 tenants by today's spend. Tells us who is affected.
- Average cost per call, by feature, weekly. Tells us when an accidental change made something expensive.
- Failed-budget-check count, by feature. Tells us when we are under-budgeted vs. when we have a feature that genuinely should not have been built.
Operating this taught us that #4 is the most useful. A "cost increase per call" line on a feature is a leading indicator that someone tweaked a prompt or upgraded a model without intending to.
What it caught#
Typical incidents:
- A retry loop with no exponential backoff that burned real money in an afternoon when the upstream had a degraded period. Caught early by per-tenant daily caps.
- A feature where an enthusiastic dev used the expensive model "for quality" without checking the cost. Caught by per-feature trend panel before it scaled to all tenants.
- A prompt that grew over six months as the team added context to it. Each addition was small; the aggregate was many times the original token count. Caught by cost-per-call trend.
- A bug where an empty-string user input was being sent to the model and the model dutifully responded with several paragraphs of nothing. We added an empty-input guard.
The ROI#
The library took modest engineering time. It prevents incidents that would have cost more than that engineering time. It has also let us confidently build with model calls in ways we would not have without it. The "we know what this costs" psychological effect is at least as valuable as the savings.
The next part of this series wraps the AI track. Or jump to the performance track if you want a different shape of essay.














