Cost guardrails for AI workflows

A model call costs cents. A bad retry loop costs hundreds of dollars in an afternoon. The first AI bill that surprised us was traced to a single feature where, on a particular error path, the code retried up to thirty times before giving up. The function "worked." The bill did not.

This essay is about the small accounting library we wrote in response, which has paid for itself many times over.

What the library actually is#

A wrapper around every model call that:

1. Counts tokens before sending.
2. Records actual usage from the response.
3. Attributes both to a (tenant, feature) pair.
4. Refuses to dispatch the call if the tenant is over budget.

About 200 lines of code total. The whole thing is a thin layer between "my code" and "the model SDK." It looks like this:

type Caller = { tenantId: string; feature: string };

interface ModelCall {
  prompt: string;
  model: 'cheap' | 'medium' | 'expensive';
  maxOutputTokens: number;
}

class GuardedClient {
  async call(caller: Caller, req: ModelCall): Promise<string> {
    const inputTokens = await this.countTokens(req.prompt, req.model);
    const projectedCost = this.priceOf(req.model, inputTokens, req.maxOutputTokens);

    if (!await this.budget.canAfford(caller, projectedCost)) {
      throw new BudgetExceeded(caller, projectedCost);
    }

    const start = Date.now();
    const result = await this.upstream.complete(req);
    const elapsed = Date.now() - start;

    await this.budget.charge(caller, {
      inputTokens,
      outputTokens: result.usage.output_tokens,
      cost: this.priceOf(req.model, inputTokens, result.usage.output_tokens),
      elapsed,
      feature: caller.feature,
    });

    return result.text;
  }
}

Five things are loadbearing in those few lines.

1. Token-count before send#

The cost of a call is roughly input_tokens × input_price + output_tokens × output_price. We can know input_tokens exactly before sending; we have to guess output_tokens, which is why maxOutputTokens is a required parameter. The projection is "input + max output;" the actual charge is "input + actual output."

2. The budget check is hard, not soft#

canAfford returns false if the tenant's daily spend plus the projected cost would exceed their cap. We throw, not warn. Soft warnings get ignored.

3. The (tenant, feature) attribution#

Every charge is attributed to both. This lets us answer two different questions:

• "Which tenants are most expensive?" (sum by tenant)
• "Which features are most expensive?" (sum across tenants by feature)

Cost-per-feature is the more interesting one. Three months in we discovered that one barely-used feature was responsible for 40 % of spend, because it ran a verbose model on a 50 KB prompt and the average user clicked it twice a session. We rewrote it with a smaller prompt and a cheaper model; cost-per-call dropped 12×.

4. Wall-time, not just tokens#

We log elapsed time alongside cost. Slow calls are not always expensive, but a feature that always takes 9 seconds is a UX problem even if it costs nothing.

5. Per-tenant, not per-user#

The budget is on the tenant (the company), not the individual user. This matches how costs are billed. It also avoids the bug where an admin user with privileges burns the user-budget of a colleague who did not consent.

What we set the budget to#

Three budget tiers, declared per tenant plan:

Two unintuitive choices:

Free tier has a non-zero budget#

We considered "free tier gets nothing." It is hostile. The 10-cent daily budget covers about 30 cheap-model calls, which is enough for a user to evaluate the feature without us subsidising thousands of calls per free user.

Enterprise has a soft cap#

The hard cap on Pro is the right default; for Enterprise, blocking their workflow because their admin forgot to update the cap on contract renewal is bad business. Soft cap, alert, conversation.

What we instrument#

A small dashboard with five panels:

1. Spend per hour, last 24 hours. Tells us when a spike is happening.
2. Top 10 features by today's spend. Tells us where the spike is.
3. Top 10 tenants by today's spend. Tells us who is affected.
4. Average cost per call, by feature, weekly. Tells us when an accidental change made something expensive.
5. Failed-budget-check count, by feature. Tells us when we are under-budgeted vs. when we have a feature that genuinely should not have been built.

Three months of operation taught us that #4 is the most useful. A "15 % cost increase per call" line on a feature is a leading indicator that someone tweaked a prompt or upgraded a model without intending to.

What it caught#

Real things from real production:

1. A retry loop with no exponential backoff that cost ~$200 in an afternoon when the upstream had a degraded period. Caught at ~$40 by per-tenant daily caps.
2. A feature where an enthusiastic dev used the expensive model "for quality" without checking the cost. Caught by per-feature trend panel before it scaled to all tenants.
3. A prompt that grew over six months as the team added context to it. Each addition was small; aggregate was 3× the original token count. Caught by cost-per-call trend.
4. A bug where an empty-string user input was being sent to the model and the model dutifully responded with several paragraphs of nothing. We added an empty-input guard.

The ROI#

We spent about a week on the library. It has prevented at least four incidents that would have cost more than that engineering time. It has also let us confidently build with model calls in ways we would not have without it. The "we know what this costs" psychological effect is at least as valuable as the dollar savings.

The next part of this series wraps the AI track. Or jump to the performance track if you want a different shape of essay.