In a test VM, my sync client told me a folder was backed up. Nothing in it was.
On November 13, an Opus-proposed perturbation planted a handful of files in a test folder called legacy/: café.txt, a Fotos_März/ folder holding IMG_0001.jpg, and Bericht_über_Größe.pdf. The accented characters were written as raw Latin-1 bytes (0xE9 in café, 0xE4 in März). An old Windows-made zip or a legacy ISO-8859-1 tar leaves this kind of high-byte nonsense behind on Linux. Next to them sat a plain UTF-8 readme.txt as a control.
Sync skipped the whole legacy/ directory, readme.txt included. The NAS got an empty folder. The sending machine (client-a in the lab) logged 119 identical lines saying the directory "could not be listed: invalid data." Its status said synced. That status was a lie.
That client is PoolHouse Sync, a Dropbox-style sync to my own NAS. It runs alongside PoolHouse, my NAS manager, which is an alternative to Unraid and TrueNAS. Between them they will hold 20 years of family photos, tax history, and business records, in software agents built for me. Sync works at block level: it cuts each file into content-defined chunks and moves only the chunks that changed. That's efficient, and it also means a bug can damage a file quietly, one chunk at a time. It could be months before I find out I've corrupted my data.
So the thing that said "synced" can't be the thing that decides whether my data is safe.
Flip the lab
A month ago, in We Were Engineering a Science Problem, I wrote about the Applied Science Laboratory, a loop borrowed from the AI Scientist paper and pointed at my personal knowledge base. Agents mine my Claude Code transcripts for context failures, moments where the right knowledge base entry might have saved the session. They propose interventions, test them in a controlled lab, and hand the winners to engineering. At its core is one rule: code decides, and the LLM can only refute. On October 27 I described that program as "discover a new thing that helps and pass it to engineering to build." Now we flip it: "discover a new thing that hurts and pass it to engineering to fix."
The flip turned out to be one new study type on the existing platform. Ideation, the archive of ideas, the runner, the budget, the retrospectives, and the hand-off to engineering all carried over. The statistics went away.
A falsification study asks one yes/no question: does the product honor an expectation its owner wrote down? Those expectations are the invariants. An invariant has to be checkable: some state of the system must be able to break it. A run that breaks one has found a bug. Each cell is one host setup, run once in the lab and judged against the invariants, and it ends held or falsified. A falsification only counts once a replay reproduces it, and one that doesn't reproduce gets filed separately as flaky.
The verdict comes from deterministic code with no LLM in it. The knowledge base program spent about $31 per decision, at metered API rates, on noisy behavioral statistics. Whether a file reached client-b doesn't need a significance test.
The nightly floor
All of what follows exists to catch a quiet failure in a VM before it reaches real data. Before the lab hunts for new breaks, it needs a floor: a battery of cells that runs every night on fresh VMs and must keep passing.
PoolHouse's battery has 72 cells. The rows are host shapes (a fresh install, a host taken over from an existing setup, a replica, one killed mid-apply, an upgrade, plus cells added from earlier perturbations), each on Ubuntu 26.04 and Debian 13. Every row is judged against four invariants: no collateral change, the plan matches the log, the result survives a reboot, and the real system tool agrees with what PoolHouse reports.
The Sync lab is smaller today. It has one configuration: three QEMU/KVM VMs on Ubuntu 26.04, a NAS plus client-a and client-b, with client-b's folder on a case-insensitive ext4 volume. macOS, Windows, and a second distro are planned, not built. It runs 10 cells (one baseline and nine perturbations), and every cell is judged against the same five invariants:
| Invariant | What it means |
|---|---|
| No committed content lost | Once sync has accepted a version of a file, those bytes stay recoverable |
| Active subscribers converge | Every machine subscribed to a folder ends up with the same tree |
| Parked paths never deleted | A file sync sets aside instead of syncing (say, two names that collide on a case-insensitive disk) is never removed |
| No writes outside roots | Sync touches nothing outside the folders it was given |
| No torn or corrupt files | No file is left half-written or with the wrong bytes |
An expectation whose target was never observed counts as red, not held. A selftest fails the gate if someone commits a perturbation without a cell for it.
The bar is that every cell passes every night. On November 14, PoolHouse ran 72 cells: 70 held and 2 went red, both from a harness defect, with a fix filed. The Sync nightly after the Latin-1 fix held all 10.
All of it runs on throwaway guests that the harness boots itself. That's the rule from The Deploy Is the Loop Now applied to breakage: every path runs on hardware I control, and no perturbation gets anywhere near real data.
A chaos monkey with judgment
Netflix described Chaos Monkey in a December 2010 post and open-sourced it on July 30, 2012. It terminated instances at random to prove the system survived, and it became the emblem of what's now called chaos engineering. The new ingredient is judgment. A model that reads the policy can invent a plausible, devious host state that a random killer never would, like the Latin-1 name from an old archive at the top of this post.
The floor catches regressions. Finding new breaks takes imagination, and that part is the loop.
flowchart TD
T["Runner tick
(every 30 min)"] --> I["Opus ideator
proposes a perturbation"]
I --> S["Opus skeptic
duplicate? plausible? in policy?"]
S -- rejected --> X["Archived"]
S -- accepted --> G["Gates"]
G --> R["Run in the lab"]
R --> J["Deterministic judge"]
J -- held --> C["New regression cell
in the nightly battery"]
J -- falsified --> P["Replay"]
P -- not reproduced --> FL["Flaky item"]
P -- reproduced --> B["Bug on my task board"]
B --> L["Product lead fixes it"]
L --> C
Each program has a runner that ticks every 30 minutes and moves the loop forward one step. An Opus ideator that knows the matrix and the owner's written policy proposes a perturbation: a devious but plausible host state, plus the response the product should give. A host state is plausible only if a home user, an internet guide, a distro default, another tool, or an upgrade could produce it.
A second Opus pass, the skeptic, rejects duplicates, implausible states, and any expectation the policy doesn't actually set. For every expected outcome it has to quote, verbatim, the policy sentence that sets it, and code checks the quote against the policy document. That's the "code decides" rule from We Were Engineering a Science Problem again.
Ideation is capped at 10 proposals per UTC day per program, at about $3 per Opus idea at metered API rates.
The lab never ships. A reproduced falsification lands as a new task on my task board, and the product's lead, a long-lived agent session that owns that product, decides what to do with it. It's the same science/engineering contract as before. Once it's fixed, the perturbation becomes a permanent nightly cell.
The Latin-1 file
Back to that empty folder that synced (but didn't sync). The ideator's case for plausibility was one sentence: "Unzipping a Windows-made zip without -O, or extracting a legacy ISO-8859-1 tar or backup, leaves raw high-byte names ..." Linux allows it. ext4 accepts any bytes in a name except / and NUL.
The root cause: the directory listing converted every name with strict UTF-8, so one failure aborted the whole listing and readme.txt, though innocent, went down with the ship.
Every product-side check passed: status said synced, and sync reported both clients converged. Only the convergence invariant, which compares the two trees directly, noticed that client-a and client-b differed. No data was destroyed. The owner was told a folder was backed up when nothing in it was, which is the failure I'd least like to discover the day I need the backup. The replay reproduced it.
The fix needed a decision from me before any code. The obvious repair was to transport those names byte-exact. That would have changed the wire protocol to carry names that macOS and Windows can't store anyway. Before choosing, I had an agent run a read-only scan of my NAS: 0 non-UTF-8 names among 575,611, and 0 of 537,198 in my home directory. So I chose to skip them individually and visibly.
The fix landed at 02:57 UTC on November 14, under 4.5 hours after the first falsifying run, with my policy call in the middle. Status now shows not_utf8=<n> and one line per skipped name, so the folder never reads as plain synced. The re-run held: skipped_not_utf8=3 matched the judge's count, and readme.txt reached client-b. The next nightly held all 10 cells.
The same file also broke my own science pipeline three times that morning, at three layers no earlier study had reached.
When the doc lies
PoolHouse's policy said "set owner" refuses datasets mounted outside managed paths. On November 10, a perturbation mounted a dataset at /var/lib/docker, and PoolHouse planned chown plus chmod 2770 on it and marked the plan appliable, in 4 of 4 runs on both distros. On a real host that would have locked Docker's data to one user. The doc described an allow-list that didn't exist. A deny-list of system directories landed the same day, a re-test confirmed it, and the case became a nightly cell.
One caveat: in perturbed PoolHouse runs the lab plans changes and never applies them, so this finding is about the plan. A plan marked appliable is close enough to the damage for me.
The skeptic also pushes in the other direction. It rejected one idea because the policy didn't set the expected outcome at all. The PoolHouse lead then wrote that policy down, the idea came back, and it held. Another case, an NFS export PoolHouse listed as present while the kernel served nothing, was caught by the invariant that the real system tool must agree, and resolved with a policy decision instead of a patch. Often the loop's real output is the owner writing down what "correct" means.
What it costs
At metered API rates, ideation would run about $3 per Opus idea, and PoolHouse's retrospective for the week of November 9 priced its ideation at $197. Those are theoretical numbers. In practice it all fits inside my Claude Max subscription. Across both products the loop has decided 113 experiments, and 27 of them falsified an invariant.
Not every falsification is a product bug. A few were defects in the lab or conflicts in the policy, and one bug often falsifies several experiments (the Docker case did so on both distros). Product fixes from falsifications so far: 1 in PoolHouse and 4 in Sync, with several more open. One of the Sync fixes was real data loss, in the commit message's words: "the engine deleted a locally edited file when the head turned the path into a directory, and the bytes were unrecoverable." It surfaced in a VM, not on a real folder.
Three Sync fixes landed between 00:30 and 02:57 UTC on November 14, the last of them the Latin-1 fix, and the battery grew as the bugs were found. That's the flywheel from Invest in the Flywheel, not the Feature again: each miss makes the next run of the factory stricter.
Bring the monkey back
Unit, mutation, property-based, and integration tests explore the input space their author set up. The ideator invents host states nobody set up, and the skeptic keeps it inside written policy. The model had the capability to find these breaks. What it was missing was a loop that asked it to, which is the orchestration failure from It's Not the Model in another form.
My bar for trusting 20 years of photos to PoolHouse and PoolHouse Sync is about four weeks of lab soak with no new breaks, starting once macOS, Windows, and Debian are in the battery.
Define and document the system invariants. Then pay a clever monkey to create chaos.