Cost-by-category dashboard — why the total is not enough

The sentence "monthly LLM cost: 12,800 EUR" is harmless until someone asks: on what? One number, one CFO, zero engineering action. That is why we built the cost-by-category dashboard — a four-dimensional matrix that breaks the total down to the point where it becomes optimisable by an engineering decision.

The four dimensions

1. Tenant — which customer's requests did we serve?
2. Use-case — site_chat, workflow_step, voice_transcription, embedding, or a custom category tag (category-checkout-assistant, category-onboarding-bot, etc.).
3. Provider — OpenAI, Anthropic, Azure, Gemini, Mistral, ElevenLabs, Hume, Deepgram.
4. Model — gpt-4o, claude-3-5-sonnet, gemini-2.0-flash, etc.

The matrix lives in the eng_cost_hourly table, keyed by (tenant_id, use_case, provider, model_id, hour_bucket). Columns: request_count, tokens_input, tokens_output, cost_eur, latency_p50_ms, latency_p95_ms, error_count.

Why the total is not enough

A concrete case from May 2026. The monthly number jumped from 11,200 EUR to 15,800 EUR in one month. The total-view said: "+41%". A weekend-built dashboard, on the following Monday, showed exactly this:

• In the tenant breakdown, one customer (call them Tenant-X) added +3,200 EUR.
• In the use-case breakdown, Tenant-X's voice_transcription spiked 8x.
• In the provider breakdown: ElevenLabs had stacked a retry loop on every transcription, double-billing every job.
• In the model breakdown: specifically eleven_multilingual_v2, not turbo_v2.

In five minutes the problem was identified; the fix was a two-line provider adapter retry-policy change. The total-view would never have solved this. The breakdown did.

The cockpit /cockpit/cost-by-category route

The dashboard sits on a pivot table where rows and columns pick two of the four dimensions and the metric defaults to cost_eur (also: request_count, cost_per_request, error_rate). Filters: date range, tenant subset, minimum cost threshold. A timeseries panel shows the top-5 categories' trend; deploy markers (see the prompt deployment post) overlay it.

The category field as an engineering primitive

The category field is free-form text, optional on every LLM call. Convention: <feature>-<step>. For example checkout-summary, onboarding-welcome, cart-recommendation. Recommended pattern: every new feature gets a new category. When an engineer introduces a new use case, the first code-review question is: "what category did you tag it with?" Without the tag we can never isolate the feature in the monthly view.

Wiring up alerts

We compare every category's average cost_per_request daily against the last-7-day median. If it triples (and request count did not fall to one-third, which would be the alternative explanation), a Slack message lands on #engine-cost. Over the last two months: 7 alerts, 5 were real bugs, 2 were expected feature rollouts (a larger model). 70% true-positive rate is high enough that the team does NOT mute the channel.

The engineering mindset shift

After the cost-by-category dashboard, the tone of code review changed. "What model is this going to use?" and "what is the expected cost-per-request?" are now standard questions. Cost used to be a CFO-level concern visiting engineers once a month. Now it is a PR-level concern touching every change. That is the real return, not the monthly number.