A few weeks ago an agent working in one of my projects needed to make a web app reachable across my home network. It reasoned, confidently and at length, about WireGuard interfaces, per-client addresses, CORS allow-lists, and port-forward workarounds. All of it was wrong. My desktop doesn't terminate the VPN; pfSense does, and every client — wired or VPN — sees one flat LAN with one stable hostname per service. The correct answer was one sentence.
That sentence was already written down. It lived in my knowledge base, in an entry that exists specifically because agents keep getting this wrong. The agent never looked, because it didn't know its mental model was stale. You don't search for corrections to beliefs you don't know are beliefs.
I've written before that most agent failures are context failures or orchestration failures, not capability failures. This was the purest context failure I'd ever logged — and it exposed a structural hole in how I'd built my knowledge layer. Everything was pull: the agent asks, the KB answers. But an unknown-unknown can't be pulled. Somebody has to tap you on the shoulder.
So we built a knowledge base that whispers.
Why whispering is hard
The obvious version of this feature is terrible. Run a search over whatever the agent is saying, inject the top hit. We measured it: 18% precision. Four out of five injections would have been noise — or worse, plausible noise. A frontier model that receives an unrequested-but-authoritative-looking pointer anchors on it. This isn't a hunch — the anchoring bias that skews human judgment transfers cleanly to large models, and the RAG-poisoning literature shows a single injected, authoritative-looking passage can override the genuine evidence a model would otherwise reason from. A wrong whisper doesn't just cost tokens — it corrupts what the model reasons from next. A confident-wrong whisper is worse than silence.
That asymmetry became the design's spine: a hard gate of at least 0.8 precision with approximately zero wrong injections before anything touched a live session, measured on an offline harness of real transcript slices — the actual wrong words real agents wrote in real sessions, including the WireGuard incident as golden case #1, plus a pile of hard negatives: moments that sound infrastructural but need no correction.
The harness then killed two ideas I was fairly sure would work:
Similarity thresholds don't gate relevance. Embedding similarity measures topical adjacency, not "this agent is guessing." Our hard negatives — an agent fluently narrating dispatch waves, a debugging session full of infrastructure vocabulary — scored higher than the golden positives. There is no threshold that separates them, because they aren't separated in embedding space.
Richer evidence made the judge worse. We gave the relevance filter the knowledge entries' actual documented facts, expecting sharper verdicts. Precision dropped. More material just gave the model more to rationalize a topical match with. I would have bet money the other way.
What actually worked: structure, not scale
The wins were embarrassingly cheap. Two structural rules did more than any model upgrade could have:
-
Cross-project only. Never whisper a project's own documentation into a session working in that project — the session-start surface already covers that. The whisper exists solely for knowledge from other domains.
-
Never whisper about a system the agent is operating. This is my favorite piece of the design. The session transcript names every tool the agent has called. An agent calling
dispatch_rolloutis operating the dispatch system — it doesn't need the dispatch orientation map; it's holding the controls. An agent making claims about VPN topology has no VPN tools to call. No tools for the domain + confident claims about the domain = guessing from priors. That's exactly the moment to whisper. Each knowledge map carries a hint naming the tool surface that "operates" it; the listener drops candidates whose tools appear in the transcript.
On top of those rules sits a deliberately mistrusted model: three independent votes, retrieve-and-cite only — the filter can return an ID from the retrieved candidates or NONE, never prose, enforced in code rather than in a prompt — and any disagreement means silence. Final harness run, through the deployed endpoint: 1.00 precision, zero false injections. Coverage is around half — the listener misses real moments, by design. Silence is cheap. Wrong whispers are fatal.
The delivery is a hook with no synchronous cost: when an agent finishes a turn, a detached worker analyzes what it just said; if a pointer survives the gauntlet, it arrives at the next turn, one factual line:
Possibly relevant map — [kb-01724] Home network & lab infrastructure
It arrives as one line — a pointer phrased as a fact, from a renderer whose tests assert it can never emit an imperative verb. The agent can pull the entry or ignore it. Stale beliefs persist across turns, so one turn of lag costs nothing.
It whispered during its own build
The day we shipped it, while I was discussing which machine should host the embedding model, the hook context of my next message quietly included the pointer to the home-network map. The agent in that session was making claims about my machines; it had no tools for them; the listener noticed. The system tapped us on the shoulder during the session that built it, about the exact domain that motivated it.
From one gate to a fleet
That's the single-KB story. What runs now is a small fleet. I keep more than one knowledge base — a personal one, a team one, and a third — and the listener fans out across all of them. Each KB runs its own copy of the gate, independently, over the same slice of transcript; the per-KB winners are unioned by a client-side arbitration step that can only suppress, never promote. It dedups and it silences, but it never elevates a candidate that a KB's own gate didn't already bless — no client-side relevance scoring sneaks back in through the union. Three hosted instances, three separate unanimous-3 votes, at most one pointer per KB per turn.
The whole thing sits behind a kill switch (KB_LISTENER_ENABLED, default off).
This is a pilot, and the honest default for an unproven channel is silence — you
opt a stack in, you don't opt it out.
I also stopped flying blind. Every listener decision now returns a compact
reason — matched kb-01724 (unanimous 3/3), or no-injection: all candidates dropped by rule-A, or best candidate not unanimous (2/3) — and the hook writes
a real-time, per-session decision log to disk. When the channel stays quiet I can
see why it stayed quiet, which is the whole difference between "the gate is
working as designed" and "the gate is broken and I have no way to tell."
Does it actually help? I don't know yet
The entire argument of this post is that context beats capability — and that argument only holds if the context actually lands.
Precision I can defend: the gate fires rarely, and when it fires it fires on the right domain. Over about three weeks and 20 sessions the system emitted 610 whispers — but 598 of those were the deterministic session-start roster pushes, the boring always-on directory of what exists. Only 12 were precision-gated anticipatory-listener fires. Twelve. The listener is strikingly sparse, exactly as designed: it would rather say nothing than say something plausible-but-wrong.
But sparse-and-correct is not the same as useful, and usefulness is the number that matters. So I instrumented it — a telemetry sink that logs every whisper and, the load-bearing signal, whether the whispered map was actually pulled afterward. Did the agent take the tap on the shoulder, or ignore it?
And then I found out I'd been measuring nothing. The consume-detection hook — the one piece that closes the loop from "whisper emitted" to "whisper used" — had been silently unwired the whole time. Every whisper read as unused. A flat 0%. Whispers weren't failing; the wire that would have told me was cut. Which is the most on-brand failure this project could possibly have: I couldn't see that my own measurement was broken, so I couldn't go looking for the correction — the exact unknown-unknown shape the whole system exists to catch. I only caught it because the zero was too clean to be real.
That's fixed now. Real usage data starts accumulating from here. So this is where the post has to stop and wait for the data:
I built an instrument to answer "do whispers help," pointed it at the system, and learned the instrument itself was broken. Fixing that is a small proof of the thesis on its own — the failure was invisible until I built the surface that made it visible. But the verdict on whispers themselves is genuinely still out, and I'd rather ship that sentence than a triumphant one I can't yet back with data.
That loop — failure logged as knowledge, knowledge wired into a surface, surface gated by an eval harness built from the original failure, harness killing the seductive designs and blessing the boring structural one — is the whole flywheel argument in miniature. The model never got smarter. The context around it did. And the next agent that starts confidently riffing about my VPN will get one quiet line of correction offered before it builds the wrong thing — whether it takes that correction is the thing I'm now, finally, wired up to find out.
Capability is the model's job. Knowing when you're wrong is the harness's. Build the part that whispers.