In March 2026, for four hours, one tenant's system prompt occasionally bled into another tenant's response. Not a flood — shards. A sentence here, a paragraph there. The bug lived in the Redis prompt-cache hash-key construction and fired so rarely that we could not reproduce it reliably for two days. This is a post about what caused it, how we caught it, and which bug class we still cannot fully exclude.
The faulty hash key
The original cache key was sha256(systemPrompt + userMessage + modelId). No tenant prefix — the assumption was that if two different tenants happened to send the exact same system prompt (e.g. the default "You are a helpful assistant"), a cache hit would be fine, because the response would be valid for both. That sounded innocent — until one tenant started dynamically injecting CUSTOMER-LEVEL data into the system prompt (a contact-information snippet configuring the support tone). The system prompt was now varying per end-user, but the cache key did not know. Worse: userMessage was a normalised, cleaned form — two different raw user messages could collapse to the same normalised string, fusing their cache entries.
Result: tenant A's system prompt, with its embedded contact text, became context for tenant B's response. The model generated valid answers in both cases, but the tone (and in a few cases the contact person's name) leaked across.
How we found it
The bug surfaced via a customer complaint: "Why did your bot call us Zoltán? Nobody named Zoltán works here." The logs looked clean: request OK, response OK, system prompt correct — nothing matched the complaint. We ran a controlled MONITOR on the Redis cache (in prod, two minutes, supervised) and saw it: two adjacent keys produced the same hash, one with "Zoltán" in its content, one without. On the OpenTelemetry trace, cache.hit=true and cache.key=<hash> together gave us the smoking gun.
The three-layer fix
- Tenant-scoped key prefix. Every cache key now starts with
tenant:<tenantId>:. This is the foundation; every other layer rests on top. Simple, idempotent, backwards compatible (old keys age out via TTL). - Per-key TTL class. Not every cache entry is equal. Static system prompts can cache for 24 hours, dynamic ones embedding customer data should cache for at most 60 seconds. The
TtlClassifierservice takes a system prompt and returns a TTL. Heuristics: if the prompt contains placeholder patterns ({{customer_name}}-shaped tokens before render), max 60s. Otherwise 24h. - Audit-on-hit. Every cache hit writes an
eng_cache_auditrow (tenant_id, cache_key, hit_at, response_size). This is not analytics — it is a security trace. If we ever suspect leakage again, this table lets us find cross-tenant collisions in minutes.
What we still do not fully solve
The bug class is called "semantic key collision": two syntactically different prompts that ask for the same answer. Normalisation sometimes (not always) hashes them to the same key. The full defence is to live every tenant in a completely isolated cache layer, even when the prompt is identical word for word. That is now our default — but the cache hit rate dropped from 71% to 58% because of it. The performance loss is the price of safety, and we accept it. Anyone telling you there is no trade-off here has not yet run a multi-tenant cache.
A post-mortem note on detection time
The gap between the first leakage event and the customer complaint was 92 minutes. The gap between the complaint and our first reproduction attempt was 18 hours, because the engineer who saw the ticket assumed it was a hallucination, not a leak. We changed the support-engineering escalation script after this incident: any report of an unfamiliar named entity in a response triggers a security review, not a model-quality review. The two are easy to confuse and the cost of confusing them is high. The new script has fired three times since March, and twice it correctly surfaced a non-issue (model hallucinations of plausible Hungarian names) and once it caught a real bug — a tenant-prefix that was dropped during a Redis cluster rebalance.