Quota events monitoring — a 429 is not an error, it is a signal

We used to treat provider rate-limit events (HTTP 429, Anthropic overloaded_error, Gemini RESOURCE_EXHAUSTED) as errors: log, retry, fallback chain, done. They even paged on-call when sustained. Six months in we realised: a 429 is not an error, it is the market's capacity weather report. Turning it into a signal — chartable, alertable, trendable — is one of the most undersold engine features.

The eng_quota_events table

CREATE TABLE eng_quota_events (
  id BIGSERIAL PRIMARY KEY,
  occurred_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  tenant_id TEXT,
  provider TEXT NOT NULL,
  model_id TEXT NOT NULL,
  event_type TEXT NOT NULL
    CHECK (event_type IN ('rate_limit_429','overloaded_503',
                          'quota_exhausted','capacity_warning')),
  retry_after_seconds INTEGER,
  fallback_succeeded BOOLEAN,
  fallback_provider TEXT,
  request_category TEXT,
  span_id TEXT,
  raw_response_excerpt TEXT
);
CREATE INDEX ON eng_quota_events (provider, model_id, occurred_at DESC);
CREATE INDEX ON eng_quota_events (tenant_id, occurred_at DESC);

Every 429 / 503 / quota-exhausted response writes a row here. fallback_succeeded records whether the router's fallback chain absorbed the incident (invisible to the user). request_category is the same field from cost-by-category — letting you link the quota event to the feature that caused it.

The real-time graph

The /cockpit/quota-events route shows two panels. First, a provider × event_type heatmap (last 24h, 15-minute buckets). Second, a timeseries plotting hourly event counts for the top-3 affected (provider, model) pairs. Vertical annotations: deploy markers (model swaps) and slo.degraded events.

The chart auto-refreshes every 30 seconds over SSE (see the useAdminEventsStream hook). On-call can keep it open in a second tab and glance at it every couple of minutes.

The 429 as a signal — what we do with it

1. Capacity planning. When a provider routinely 429s for a large tenant, the answer is not retry but a dedicated throughput plan (OpenAI Enterprise, Anthropic Workspaces). The graph shows when that conversation is worth starting.
2. Provider-mix calibration. When Azure OpenAI's gpt-4o consistently 429s while OpenAI direct does not, we rebalance the request mix. The 429 rate inverts within a week.
3. Tenant-level soft throttling. When one tenant consumes 80% of the quota, the rest slow down. RouterPolicyService activates a tenant-level priority queue; the heavy-user tenant's requests get routed to fallback providers first.
4. Provider capacity warning. Some providers (Anthropic in particular) send an X-Capacity-Warning header before the 429 hits. We record those too (capacity_warning event_type) and pre-empt the route switch before the user sees anything bad.

Wiring up alerts

Not every 429 is alert-worthy. Threshold: when a (provider, model) pair generates 50+ events in 5 minutes AND fallback_succeeded=false exceeds 10%, message lands on Slack #engine-ops. fallback_succeeded=true does not page because users are unaffected — the router absorbed it. With that tuning we average 2-3 real pages per week instead of 20 per day.

One real-world day

May 28, 2026, Thursday morning. OpenAI gpt-4o p95 jumped from 800ms to 2400ms and 429s started flowing. The eng_quota_events chart showed the incident started at 06:14, confined to eu-west-3. Tenants on Azure OpenAI were unaffected. The router's automatic shift to Anthropic Claude 3.5 Sonnet completed within ten minutes (fallback_succeeded=true rate 94%). User complaints during the incident: 0. status.openai.com posted at 09:00. By then the router had been handling it for two hours. That kind of resilience is impossible when a 429 is treated as an error.