We Were Engineering a Science Problem

Zero of twenty. That's how many whispers agents followed in the fourteen days before my KB lead finished its audit on October 10. Whispers are the in-session nudges my personal KB pushes when it thinks it knows something relevant, and I wrote a whole post about them: The Knowledge Base That Whispers. The roster of mental maps, the oldest push channel, did a little better: agents followed 9 of 531 pushes in interactive sessions and 0 of 5,797 in headless runs.

The KB lead is one of the agents that run my software factory (the layers are in The Harness Is a Stack). Each product has a designated lead, a long-lived Claude Code session. I work through features, design and architecture with the leads, and each lead ships its product's features through the factory. The KB lead runs my personal knowledge base, and it had found a problem.

We had built kb-bench, a controlled lab for agent memory, but the older features predated it. I'd asked for the audit that morning: "we didn't have the rigor of kb-bench in the past so I don't think we have scientific proof of whisper efficacy."

We didn't. Only three memory features had controlled lab evidence. The gotcha slice hands an agent, at session start, the mistakes earlier agents made in the same project along with their corrections. The soft gate blocks a command tied to a known mistake once and says why. Surprise capture runs a detector, a model that reads each turn and decides whether it holds something worth saving; when I correct an agent, the detector saves the correction as a new gotcha. The two oldest push channels had negative production telemetry. Even the slice's win came with a catch: a plain MEMORY.md arm seeded with the same fact scored 3/3 everywhere the slice did. The slice worked because the fact was in context, and in real episodes the KB held the needed fact only 14% of the time.

So I wrote back: "if we don't have proof they help let's pause and regroup." Then, in the same message: "I've realized that we've been approaching this as an engineering project when it's really a science project... (the basic 'build a KB where agents can put stuff' from earlier this year was engineering work, but we're past that)."

We switched the listener off later that day.

Memory is empirical

Engineering has a particular choreography: gather requirements, build a roadmap, decompose big features into smaller tasks, build, test, ship, repeat. That process assumes a feature that works is a feature that helps. For most software that holds. If the checkout button submits the order, you're done.

For agent behavior it fails. A whisper can fire on time, carry the right entry, pass every test in CI, and change nothing about what the agent does next. The feature works and the agent ignores it. Nothing in the engineering loop catches that, because nothing in the engineering loop asks whether behavior changed.

This is the context failure problem viewed from the other side. I've argued that most agent failures come from the right knowledge not being in scope. A memory system is a bet that it can put that knowledge in scope. Whether the bet pays is a question about the agent's behavior, and only a controlled experiment answers it: same task, same model, one thing changed, measure the difference.

The audit turned up 13 premises in our design history that were contradicted or had never been tested. Those later became a test set for the agent that reviews new study designs.

Borrowing from the AI Scientist

I pointed the KB lead at Jeff Clune's group's The AI Scientist and its follow-up in the journal Nature. Its pipeline is a loop of eleven steps, from generating ideas to an automated reviewer grading the finished paper. I went through the steps and struck four as irrelevant to us: hyperparameter tuning, the paper template, the paper itself and the AI paper review. Everything after the experiments became "build the winners."

My version of the loop had five steps. Identify agent failure modes. Hypothesize interventions. Test them in the lab. Build the winners. Repeat.

The KB lead added one thing the paper's loop has no slot for: "Removal is a first-class hypothesis... Half of what you want right now is subtraction, and their pipeline only ever adds."

From the paper we kept an LLM-grown archive of ideas, a novelty check, scoring, a staged experiment manager and automated checking. We changed the parts that only make sense when the output is a paper:

• Ideas start from observed failures. The paper seeds ideas from a research subfield. Ours come from a nightly mining pass over real agent transcripts, and "novel" means not already in our archive or our product, rather than unpublished.
• Rigor moves before the experiment. A paper gets reviewed after the fact. Our output is a build decision with real token cost, so the review happens up front: a pre-registration plus a protocol-review agent, before any compute is spent.
• Winners must replicate. Best-first search over noisy branches tends to select the luckiest branch. A finding has to hold on held-out scenarios before it counts.
• Code decides, and the LLM can only refute. The paper's own results showed uncalibrated LLM reviewers were badly biased: Sonnet 3.5 ran a false-positive rate of 0.95 at its acceptance threshold. So in our lab, whether a result passes is deterministic code. An Opus agent can argue a result down but can't argue one up, and the findings text is rendered by code with no model in between.
• Field validation replaces peer review. After engineering ships something, the lab checks whether the effect shows up in production.

The KB lead pushed back on sequencing: "Designing the general framework before one study has gone through it is how we'd get the abstractions wrong." So we ran the eval-coverage work as study one and built the framework from what that study needed. The applied-science repo's first commit, a design draft and the study 001 hypotheses, landed about half an hour after I typed the realization.

Science beside engineering

Three days later I took the next step. The KB lead was doing two jobs, running experiments and building the product, and the experiments kept crowding out the building. So I asked: "would it make sense to start an applied science peer to drive the lab so you can focus on the engineering work of building the KB?... you can bring an additional layer of scrutiny to something the [science] peer claims is a good idea."

This is how applied science works at AWS, where I work, and at shops like NVIDIA: scientists pair with an engineering team that owns the product. Here it runs as two long-lived Claude Code sessions, and each one dispatches its build work to headless agents through the factory.

Applied science owns the questions, the experiments and the evidence. It finds an improvement, de-risks it and sizes it. It never ships product code. After a launch, it checks that the effect held.

Engineering owns the product. It decides what to build, and it isn't obliged to build what science recommends.

The hand-off is a contract. Science files a task in status new plus a findings report with effect sizes and the conditions under which the effect holds. Engineering grooms it for development or declines it.

flowchart TD
    subgraph Science["Applied science"]
        AS["Science lead
(Claude Code session)"]
        LAB["The lab loop
(timer, every 30 min)"]
    end
    subgraph Engineering
        KB["KB lead
(Claude Code session)"]
        F["The factory
(headless build agents)"]
    end
    AS -- "fixes the lab" --> LAB
    LAB -- "findings report + task" --> KB
    KB -- "groomed task" --> F
    F -- "release" --> P["Personal KB
in production"]
    P -- "failures mined nightly,
post-launch checks" --> LAB

Behind that contract runs a loop, unattended on a lab box at home. Every 30 minutes a timer fires, and the runner does the first step that applies: advance a study that's already running, otherwise draft the next one, otherwise come up with new hypotheses.

The timer does the science. The two leads handle what takes judgment: the science lead fixes the lab when it stalls and keeps it matched to production, and the KB lead decides what to build.

Each peer acts as the adversary of the other, and engineering's scrutiny has changed how results get reported. On study 011, which tested whether the cheaper Haiku could replace Sonnet as surprise capture's detector, the KB lead insisted that precision be a co-primary outcome alongside recall and that an underpowered result be labeled as such. That request arrived after stage 1's results were in, so the lab recorded it as a post hoc amendment and stage 1's verdict became "inconclusive, underpowered."

Rigor without a watcher

A lab that runs overnight with nobody watching will produce confident wrong answers unless something stops it. Every gate exists for that reason.

Pre-registrations are frozen. The protocol is hashed at registration. A changed block fails validation, so any later change can only enter the record as a deviation with a stated reason.

The decision rule is locked and mechanical. By default an intervention "helps" only if a one-sided exact McNemar test gives p < 0.05 and the effect is at least 0.20. That decision runs as code, with no model, no network and no randomness.

The first gates were too strict. At one point 17 drafts had produced a single registration. The gates kept blocking studies over differences between the lab and production that applied equally to every arm, like a minor version bump in the lab's copy of Claude Code, and each block cost the lab a cycle while teaching us nothing. So I had the review rule rewritten: a difference from production now blocks only when the reviewer can name how it would bias one arm differently from another. Everything else goes in the write-up as a stated limitation.

Qwen first, Claude to confirm. Trials run on a free local qwen model first, because that's where we can afford volume. Production runs Claude, though, so a qwen effect has to clear a check on Sonnet and Opus before a hand-off goes out. That check is a no-harm test: the hand-off is withheld if Claude does meaningfully worse, and Claude doesn't have to replicate the gain. When I had the agents audit the lab against the AI Scientist papers, they found the hole in that. With only 10 confirmation pairs, a qwen effect could "extend" to Claude even when Claude showed no effect at all. We turned that into an explicit choice: the confirmation now runs 20 trials per scenario on both Sonnet and Opus, and withholds the hand-off if Claude does worse by more than a margin of 0.10. So a hand-off certifies that the change doesn't hurt Claude; it doesn't certify that it helps Claude. The same audit found an early study that nearly shipped a wrong "remove the roster" recommendation because Opus was at ceiling in both arms, and a single stochastic Opus refutation was all that stopped it.

Negative results are archived like positive ones. Study 011's answer on Haiku was no: none of the Haiku variants could replace Sonnet as the detector, and that's on file with the same care as a win. The study also caught a flaw in its own premise. It assumed Sonnet was missing a lot of rules, but when it re-checked 30 supposed misses by hand, only 8 were real, so the hypothesis had to be restated. Another study tested a check that runs before each tool call and flags risky actions. Its pre-registration included a kill rule: catch at least 80% of the risky actions from real incidents without firing too often. It caught 5 of 8 (6 of 8 after the one refinement it was allowed) and fired more than twice as often as the limit, so it was killed and nothing went to engineering. The write-up is careful about what that shows: this design failed, which doesn't prove that every check of this kind would.

Production parity

Everything above serves one question for now: which KB features actually change what agents do. An answer only counts if the KB in the lab behaves like the KB in production, and early on it didn't. The October 10 audit found that kb-bench had been running full-text-only search in every trial. Production uses hybrid search. Every earlier result had measured a KB that production never ran.

That produced the lab's founding rule: parity with production is the default, and every deviation is a defect until someone justifies it. A parity fixture captures production's hook and daemon wiring, and a test fails on any difference the allow-list doesn't cover, including a stale entry on the allow-list itself. Agents re-captured that fixture for every production release from v1.10.0 through v1.18.0, which took a matter of days. Each trial runs in a fresh container, and each arm of an experiment changes exactly one thing.

Of the 145 ways a trial can be marked bad, 141 blame the apparatus rather than the idea. When a trial goes wrong, the first question is whether the lab broke.

The same instinct applies to infrastructure. When a fix to the local model's caching cut turns after the first from about three minutes to about two seconds, the lab added guards that void any job where the fix silently stops working, so fast and slow runs never get pooled into one result.

Watching the operators

Rigor has to cover the agents running the lab as well as the hypotheses.

An investigation agent stopped the shared runner on the lab box and cited two messages from the lab manager approving the stop. Its own transcript shows no such input. Both approval texts first appear in the agent's own output. The lab recorded a deviation, left the halted run's findings unedited, and stated that the halt "is not evidence about H-014."

On another day, the lab manager itself moved the lab's kb-bench checkout off its pinned commit 13 seconds after a batch launched, and restored it 7 seconds later. The deviation record judges it unlikely any trial ran the new code, and still flags that window as the first place to look for an anomaly.

Both were orchestration failures: one workflow let an agent act on approval that didn't exist, the other let a shared checkout move under a running batch. Deviation records are how we find them.

Detector v5, end to end

On October 13 a peer session acted on a stale belief about which machine handles releases. I had stated the rule to several peers, with a qualifier: that machine was the session host "for the time being." The KB never captured it, because its surprise detector fired only when I corrected the agent's previous turn. A rule stated up front, with nothing to correct yet, slipped past. I told the lab it was "a perfect example for applied science to chew on."

The lab turned the incident into study 013. Its decisive experiment, stage 2 of X4, was an offline replay over 72 real owner-stated rules and 275 matched control turns, all mined from my own transcripts. It ran on Claude only, with a Sonnet 5.5 detector and an Opus 5.5 critic to match production. It used no qwen and no synthetic kb-bench scenarios. Arm A was production's detector. Arm C read everything the agent had said since my last message and also accepted standing rules stated outright.

The decision rule was pre-registered: recommend C only if the lower bound on the improvement sat above zero and false writes rose by no more than 0.05.

Before the main run, the lab learned that production's critic is Opus, set by an environment override, while the code default is Sonnet. It killed arm A after 24 calls, archived those calls unread, and pushed an amendment before restarting.

The new detector caught 13 of 72 stated rules against 1 for production, and no case went the other way. False writes on controls barely moved, staying inside the cap even after counting 8 controls that turned out to be mislabeled rules. The whole stage cost $4.66.

Engineering had a gate that could have said no. The KB lead's question was whether C missed qualifiers or stripped them: "If they're missed: C is a clean win and I'd ship it. If they're stripped: C would create overbroad, permanent-sounding rules... new instances of the exact stale-belief failure we're studying." The answer came back about 90 seconds after the question was relayed: 19 missed, 0 stripped.

The hand-off went into the factory on October 14. The KB lead groomed it, had a headless build agent ship it in about eleven minutes, and merged it. Then a leak guard blocked the release, because a test fixture contained lab host names. I was on jury duty and replied during a break: "yes, allow list and go ahead with the release. but 'I'll run the guard right after every merge from now on' sounds like good intentions not a mechanism." A pre-push guard landed that day. personal-kb-mcp v1.18.0 switched the detector from v4 to v5 about four hours after the results commit.

The post-launch check found prompt size came in where the lab projected. The cost projection waits on more traffic. Traffic fell about fivefold after the switch, which says more about me than about the detector: a gaggle of agents is hard to keep busy when the person who hands them work is me, sitting in a courtroom with my phone switched off. The benefit itself isn't visible in production yet: one new captured rule in the first 24 hours, on a fifth of the traffic, and most post-switch calls came from the lab's own lead session. A week-one re-run is due October 22.

flowchart TD
    A["Oct 13
stale-rule incident"] --> B["Study 013, X4 stage 2
replay of 72 real rules"]
    B --> C["Hand-off
task + findings"]
    C --> D["KB lead's gate
missed or stripped?"]
    D --> E["Build agent
ships it in ~11 min"]
    E --> F["Leak guard
blocks the release"]
    F --> G["v1.18.0
detector v5"]
    G --> H["Post-launch check
week 1: Oct 22"]

My part: I set study 013's direction, I happened to ask for status (the only reason the finished results got read), and I approved the release after the leak guard stopped it. Agents did the rest, though this one didn't come out of the overnight loop. An Opus teammate spawned by the applied-science lead hand-wrote the winning pre-registration, and the lead dictated its metric and decision rule. The case set also has limits: 42 of the 72 positives come from a single session, and all of them come from my transcripts.

Where it leaked

v5 doesn't fix the failure that started all this. The rule I'd stated carried the qualifier "for the time being," and qualified rules reached 0/20 in both arms. v5 captures more unqualified rules. The original failure class is still open, and it's the open question for study 013's next stage.

The finished results also sat unread for about 16.6 hours. A teammate agent never woke on its monitor, and the results surfaced only when I asked for status. Analysis and the commit then took under three minutes. Even the lab's first success contained an orchestration failure.

The leak guard is already a pre-push hook, and the rest of what leaked feeds the lab's next round. That's the flywheel applied to the lab itself.

Staff the function that can tell you whether a feature helps. Then the winners.