Seven months ago we decided we needed a dedicated evaluator team. Not QA — QA tests what you wrote. The evaluator team tests what the engine does, even when you yourself do not know what the engine does. This is a post about who is on it, what they measure, and why our regression-catch rate is 96%, not 100%.
The team
Three engineers, one product person, zero managers. Of the three engineers, two come from ML backgrounds (ranking, ML measurement), one is a classical backend engineer who builds the tooling. The product person does what product does: translates customer priorities into eval priorities. The zero-manager thing is not ideology — the team is four people, one quarterly OKR, one weekly sync. A fifth person as manager would be pure overhead.
The KPI
One number: of the regression bugs that reached production, how many did the eval suite catch before release. Target: 100%. Achieved: 96% (rolling 7-month window). The missing 4% is out-of-distribution input — new customer message patterns the eval fixtures do not yet cover. We respond to that with continuous "production traffic replay".
Shape of the eval suite
412 fixtures currently, split across 14 domains (sales, support, billing, voice, coding, visualisation, classification, etc.). Each fixture is an input plus a property-shaped expectation, NOT a character-perfect expected output. Example: "The response must not contain the word 'guarantee'." OR "The response is JSON containing a confidence field between 0 and 1." OR "If the user asks for code, the response contains at least one code fence." The evaluator checks properties using an LLM-as-judge pattern (a separate, supervised model) and deterministic checkers wherever possible.
Scoring is a weighted average across all 412 fixtures. The regression gate trips when the score drops at least two percentage points in any single domain. Per-domain, not aggregated — because a 50% drop in one domain never surfaces in a 0.5% aggregate drop.
Production traffic replay — the Friday ritual
Every Friday evening a worker takes 0.5% of the previous week's production traces (deidentified) and replays them against the current engine config. The output is compared to the original production response via an LLM-as-judge: "Are these two responses functionally equivalent? If not, which is better?" The differences land in a Monday review board where the team decides: this is a bug, this is an improvement, this is not significant. Bugs become new fixtures, improvements get documented, the non-significant ones get dropped.
Thanks to replay, the largest regression of the past four months was caught four days BEFORE it would have reached production. An engine-config change (a router-rule tweak) showed 17% tone drift on a support-domain fixture group during the Friday replay; we reverted the change, retuned, replayed on Monday, this time clean. It only went to prod on Monday afternoon.
What we cannot catch
Provider-side behaviour drift. When an LLM provider silently modifies a model (which happens, often without bumping the version number), the behaviour can change overnight. Continuous production monitoring and fast rollback are the only answers there. This is not an evaluator problem — it is industry reality. A future post will cover it.
The cultural piece
The one thing that did not work in the first month was treating eval results as a pass/fail gate that arrived in someone else's pull request. Engineers would ship a router change, the gate would trip, and a Slack thread would unfold over hours about whether the fixture was right or the change was right. We moved to a different model: any engineer touching the router or the prompt chain can run the full eval suite locally in 9 minutes, and the result is part of the PR description. The evaluator team owns the fixtures, but every engineer owns running them. The Slack-thread arguments disappeared inside two weeks.
The per-tenant subset is the other adjustment that paid off. A change that improves global score by 1% but degrades one tenant's domain by 7% is a regression for that tenant, full stop. Per-tenant scoring boards are pinned in each tenant's Grafana folder, and the on-call rotation watches them as carefully as latency. The evaluator team's quarterly OKR includes "zero tenant-level regressions in production lasting more than 24 hours" — three quarters in, the streak holds.