Cost is an engineering concern at the engine, not a finance problem. If a developer looks at a slow request's trace and cannot see what the request cost, then cost lives on someone else's desk, and someone else will notice the drift too late. This post is about how we put per-request cost onto every OpenTelemetry span, what we learned doing CFO-grade reconciliation quarterly, and why the monthly per-tenant view never matches the sum of trace costs.
Per-span price
Every LLM call is its own OTel span. The span attributes carry llm.tokens.input, llm.tokens.output, llm.model.id, llm.provider, llm.cost.eur. The llm.cost.eur field is computed live: token count × current price-per-token from the eng_pricing_table row. This matters: we do not write "cost to be calculated later" on the span, we write the cost as of the moment of the request. A Grafana dashboard then shows per-tenant hourly cost, top-N most expensive requests, and a per-model cost breakdown — all from spans alone, no offline batch job.
The cost-attribution-collector is a BullMQ worker that reads fresh spans from Loki every 30 seconds and writes aggregated rows into the eng_cost_hourly table. Aggregation keys: tenant_id, model_id, hour_bucket, request_type. Reaggregating the same request does not double-count because there is a dedup_key = trace_id + span_id. The worker also writes a tool.cost.eur attribute for tool-call sub-spans (e.g. RAG retrieval, web search), so a chat session's cost is the sum of every sub-span, not just the top-level LLM call.
The pricing-table refresh job
The pricing table auto-refreshes every 6 hours via a pricing-refresher worker that reads each provider's public pricing API. Some providers do not publish a pricing API (lookup table embedded in code, refreshed manually each month). The refresh is transactional — either every new price is written, or none. Historical prices live in eng_pricing_history, so if anyone recomputes the cost of a six-month-old request, they get the price as of then, not today's. This is an audit-level requirement at several customers, and it cost us a week of engineering time to backfill the pricing-history table from provider release notes before we could enable it.
Why the monthly view does not match
The monthly per-tenant view says tenant A consumed 4,821 EUR. The trace-level sum says 4,967 EUR. The gap is 146 EUR, about 3%. The cause is sampling. We do not export every trace to Loki, only a representative sample (1:10 on standard requests, 1:1 on errors and slow requests). The aggregation runs on the full event stream (at the Kafka layer), not on sampled traces, so the monthly view is the more accurate figure. The trace-level sum always drifts a little — we surface it on Grafana with a trace_coverage_pct field so engineers do not get confused. A field engineer once spent two days chasing a phantom 4% discrepancy that turned out to be exactly this sampling effect; the field has been on the dashboard ever since.
The quarterly CFO-grade reconciliation
Every quarter a reconciliation run executes: per-tenant billed amounts are read from Stripe billing records and compared against the engine's own eng_cost_hourly aggregates. Any tenant where the gap exceeds 0.5% gets manual review. Across the last four quarters we found bugs three times. All three were edge cases in the pricing-history table (requests landing between an old and a new price revision). All three are fixed. Finance now calls this process "audit-quality", which is the highest praise an engineering team can ever get from a finance team.
The deeper point is that cost belongs on the same surface as latency and errors. A request that took 8 seconds and cost 0.18 EUR is a different incident than one that took 8 seconds and cost 0.002 EUR — same latency, very different problem. The trace-level cost attribute is what lets the on-call engineer tell those two stories apart at 3am without opening a billing system tab.