Building the self-learning loop — four phases, one funnel

The self-learning loop is not an LLM feature. It is an industrial pipeline that turns user conversations into quality-improving prompt changes. The engine runs it in four phases, and the cockpit's LearningLoopFunnel widget displays exactly those four phases on a horizontal bar chart. This is a post about the phases, the numbers to watch, and how the loop closes back to the prompt deploy markers.

Phase 1: conversations

Every completed conversation (site_chat, voice, workflow) is an eng_conversations row. The funnel's first number is the raw material we start from. An average day: ~12,000 conversations across the platform.

Scope narrowing starts here. Not every conversation gets scored immediately. Two filters: at least 2 user messages (a greeting is not relevant) and not internal test traffic (rows with is_internal_test=true are skipped).

Phase 2: scored

The remaining conversations are evaluated by ConversationScoringService across seven dimensions: semantic_accuracy, tone, helpfulness, brevity, factual_correctness, hallucination_risk, safety_compliance. Each is a 1–5 scale; a judge LLM (Claude 3.5 Sonnet by default, deterministic temperature=0) reviews the whole transcript.

The funnel's second number is the count of successfully scored conversations. Average day: ~9,800 (the remaining 2,200 are simple greetings or internal tests).

Phase 3: flagged

From the scored set we select the ones where at least one dimension is ≤ 3.0. Those become the "flagged" rows. Average day: ~340 — about a 3.5% flag rate. Flag severities: low (one dimension at 3.0), medium (two ≤ 3.0), high (one ≤ 2.0 OR safety_compliance ≤ 3.0).

high flags land immediately on the /cockpit/admin-trainable dashboard where an engineer (or the review agent — see below) can react.

Phase 4: resolved

Resolved is where actual learning happens. Three actions:

1. Prompt edit. The engineer reads the flagged conversation's transcript, identifies the prompt failure, deploys a new prompt version. New deploy marker, measurable score impact.
2. Add to eval set. The conversation joins the tenant's eng_eval_sets table; every future prompt change is regression-tested against it.
3. Mark as expected. The conversation was flag-worthy but the behaviour was expected (e.g. "a novice user asked a confusing question, the agent correctly asked for clarification, the brevity score is low but acceptable"). An expected_low_score label, and it is excluded from the next scoring pass.

The funnel's fourth number: this many flags the team (or the auto-review agent) resolved within the observation window. An average week: ~1,800 flags, ~1,100 resolved (61%). The remaining 39% sit in queue.

The horizontal bar widget

LearningLoopFunnel is an SVG bar chart, four rectangles in a row. Rectangle length is proportional to the number. Label above each rectangle shows phase name and count. The gap between consecutive rectangles is "leakage" — that many conversations dropped at the phase transition. Hover reveals the reason (e.g. "2,200 conversations skipped: not eligible").

The widget lives at /cockpit/learning-loop but a mini version sits at the top of the dashboard so every cockpit visit starts with that glance.

How the loop closes onto deploy markers

The loop closes when a flag resolution → prompt edit → new prompt deploy → deploy marker on the chart → score trend measured after the marker. If drift is positive (improvement), the deployment gets a validated status. If negative, an automatic revert recommendation. validated prompt versions enter an internal changelog and, after human review, become candidates for similar tenants' prompts.

That last step — cross-tenant learning transfer — is what we call Phase 8 and are building now. The first seven phases work, and based on ~1,100 daily flag resolutions prompt quality is measurably improving: average semantic_accuracy was 3.8 in January 2026 and 4.3 by mid-June. That is not a small jump. That is two years of engineering distilled into a single number.