It's Still Not the Model

Two months ago I made a claim I couldn't measure. I argued that when a capable model fails on a task you actually needed done, the cause is almost never capability — it's one of two things: the model didn't have the right context, or the workflow wouldn't let it use what it had. Context failures and orchestration failures. I argued it in theory, with anecdotes. Now I've measured it.

So I ran it. Two small local models — qwen3.6:35b and gpt-oss:20b — that had struggled as engines under Claude Code, a big general-purpose harness, in my headless dispatch system completed 58–75% of a set of real coding tasks when I dropped them into a lean harness I built from scratch. The catch, which I'll get to: those fixture tasks are much smaller than dispatch-scale work, so this isn't a clean rematch. But the same weights that reached almost nothing under the big harness reached most of these under the lean one. Same models. Different machinery around them. The capability didn't move. What moved was whether anything could reach it.

That's an orchestration failure made visible. And the variable that made it visible is the one almost nobody isolates: the harness.

A harness you can hold still

I've been building a minimal agent harness in Rust — the loop that turns an LLM into an autonomous agent: prompt assembly, tool dispatch, model I/O, state, and the safety rails around all of it. The stated goal is to learn the design space by building one, with a concrete use case (another build engine for Agent GTD) keeping me honest. The unstated bonus, which I didn't fully appreciate until session four, is that a harness you build yourself is a harness you can hold still.

That's the property that makes it an instrument. The loop assembles one byte-stable system prompt and hands the model exactly six tools: read_file, list_files, edit_file, run_command, run_checks, and finish. Every model runs through identical machinery. The trick that makes "identical" true is an anti-corruption boundary drawn high: a ModelBackend trait where each adapter owns all the wire translation and hands the loop a normalized AssistantTurn. The loop never sees a provider-native shape. Anthropic's ID-tagged tool_use blocks and Ollama's ID-less tool calls come out the same door.

I got to watch that boundary pay off for real. The day I merged the Ollama adapter, an entirely different wire protocol — no tool-call IDs at all, so you match calls to results by name and order — drove the unchanged loop. One adapter even serves both localhost Ollama and the cloud endpoint, because they're wire-identical; only the auth header differs. When you can swap the model backend and change nothing else, the model becomes the only variable in the experiment. That's the whole point.

Why "58%" isn't a lie the model told me

Before any number in this post means anything, you have to trust the word "completed." Models hallucinate "done" confidently, and small local models do it more. So the harness never trusts the claim.

finish(done) is a claim. When the agent calls it, the harness runs the checks itself — it runs cargo test — and only if they pass does it construct a Done value. And here's the part Rust gives you for free: a Done is not constructible without a CheckReport or an explicit no-checks marker. The type carries its own evidence. A criterion can be marked Verified(Evidence), and only the harness ever writes that state — the agent literally cannot. A red check isn't a failed completion — the loop feeds the failure back and the model keeps going.

So when I report a pass, it's mechanically green, verified by the harness. When I report a failure, it's the loop bounding a model that couldn't converge — never a false "done." Across every verified trial in the metrics run, there were zero false dones. Every pass was cargo test green. Every failure was a model that couldn't converge.

(One of the build agents, unprompted, invented a private FinishClaim type the spec never named — a pre-verification view of the agent's assertion, separate from the verified Done — because the type system rewarded the distinction. I didn't ask for that. The "illegal states unrepresentable" ethos had propagated into the agents building the thing. More on that at the end.)

The matrix: same loop, four models

Here's the core evidence. Same loop, byte-identical prompts and tools, the same four fixture crates, k=3 trials each, max 12 iterations, every "done" verified by the harness running cargo test itself.

Two wrinkles jump out of the data, and I'd rather name them than let you find them.

First, k=3 means real variance, even at the frontier. Sonnet landed 11/12 in this run but 12/12 in an earlier pre-metrics run — combined 23/24. Don't over-read a single-run pass^k difference; the noise floor is not zero.

Second, StoppedWithoutFinish — the model deciding it's done without calling finish — showed up at both ends of the capability spectrum, on sonnet and on gpt-oss:20b. That's a protocol-discipline failure — the model ignored the harness's contract for signaling completion — not a capability gap. The eval can measure it precisely, which is the point of having an eval that verifies mechanics instead of vibes.

The harness was the variable all along

Now the payoff, fenced with every caveat it needs to survive.

Under this purpose-built harness — six tools, a lean byte-stable prompt, mechanical verification — qwen3.6:35b and gpt-oss:20b completed 58–75% of the fixture tasks they'd failed to finish as engines under Claude Code. That is the definition of an orchestration failure from the first post: the capability was present; the workflow wouldn't let the model reach it. Change the workflow, and the capability shows up.

The caveats, stated plainly, because a claim this convenient deserves the scrutiny:

These fixtures are much smaller than dispatch-scale tasks. A fixture crate with a planted bug is not the same shape of work as a real GTD item, so this is not an apples-to-apples rematch — the local models didn't suddenly become viable dispatch engines. The think-level was uncontrolled between the two local models: gpt-oss:20b ran with think=high, qwen3.6:35b ran with think unset. So 20b outscoring 35b is suggestive, not a concluded capability ranking — I changed two things at once and won't pretend I didn't. And the whole thing is directional evidence for the harness-as-variable thesis, not a controlled harness-vs-harness study. I didn't run the big models through the small harness and the small models through both under a proper control matrix. That's a different experiment.

What I'll defend is the direction, and it's not subtle: the same weights that reach almost nothing under one harness reach most of these tasks under another. If capability were the binding constraint, that couldn't happen. It happened.

What the eval refused to prove

The most useful thing the eval did was refuse to cooperate with a story I wanted to tell.

I'd built three deliberately subtle bug fixtures to stress the frontier tier: a < vs <= boundary error in an interval merge, a missing recency-refresh in an LRU cache, a UTF-8 char-boundary panic in a text preview. Genuinely nasty bugs — the kind that survive a code review. Sonnet swept the suite 12/12, subtle bugs included.

That's the eval telling me my difficulty knob was the wrong knob — Sonnet isn't superhuman. A failing test does the bug localization for the agent — it points a bright arrow at the broken behavior. Subtlety of the bug barely matters once the test hands you the repro. The real difficulty lever is task shape: withhold the failing test, or hand the agent a prose bug report and make it find the repro itself. Bug subtlety saturates; task shape doesn't.

I'm reporting the eval's ceiling as a known limitation because that's what an instrument is for. It steers the roadmap. I don't hang it on a wall as a scoreboard. The same run surfaced the eval's own blind spot — failed trials are opaque, because there's no trajectory persistence, so a MaxIterations failure is just a shrug with no story. That opacity is now the data-driven argument for what I build next: durability and run-record wiring, so a failure explains itself. The eval found the hole in the eval. I'd rather that than a clean number I couldn't interrogate.

The factory that built the instrument

Here's the part that closes the loop back to the other posts. I didn't hand-write this harness. It was largely built by the orchestration practice those posts describe: 15-plus headless dispatch runs across the project, with an AI lead reviewing and merging every branch on the fully integrated tree — zero quality misses after a single typos miss in session two. The core of the v0.1.0 release landed as a three-wave overnight build, seven dispatches, roughly an hour of wall-clock while I was asleep.

The reason I trust these numbers is that the same verify-don't-trust discipline runs all the way down. The very first commit to this repo wasn't harness logic — it was the quality-gate scaffold: fmt, clippy at pedantic, nextest, coverage held at 95%, machete, deny, typos, gitleaks, enforced at commit and push. Gates before code, so the agents could run wild safely. Around 230–260 tests and ~98% coverage the whole way through. rustc itself is a gate — the borrow checker, exhaustiveness, no-null, Result-based errors give you for free what a pile of Python tooling has to buy.

And the releases are cut at capability boundaries, not engineering milestones. v0.1.0 was "the harness autonomously fixes a failing test in a real crate and verifies cargo test green itself." v0.2.0 was "the same loop, unchanged, completes the suite on non-Anthropic models, measured against verified checks." The test I apply is: what does the first CHANGELOG entry want to claim? If the answer is "the spine runs," it's not a release. If it's a capability, it is. That FinishClaim type from earlier is the tell: I never asked for it, but the illegal-states-unrepresentable ethos the factory runs on had started running in the agents too — they reached for the same distinction the harness's own design rewards. The instrument that measured orchestration was itself built by orchestration, down to the way its builders think. That's the argument itself, load-bearing.

Control for the harness

So here's the actionable version, and it's just the old diagnostic with data behind it now.

When a model fails on something you needed it to do, ask three questions in order. Did it genuinely lack the capability? Least likely — reach for it last. Did it lack the right context? Engineer the information layer. Did the workflow force it below its capacity? Engineer the workflow. Two of those three are fixable today, without waiting for a lab to ship anything.

The harness is the third one, and it's the variable almost every "the model can't do X" claim never isolates. People swap models, tweak prompts, adjust temperature — and hold constant the one thing that most determines whether the model's capability is reachable at all. Then they conclude the model can't. What they've measured is the harness.

I found this out by building one small enough to hold still. You don't have to build yours in Rust — you don't have to build a harness at all to get the lesson: before you conclude a model can't, control for the machine it's running inside.

And when you find the edge of that machine — the place where the tools stop and the capability is stranded just past them — don't route around it one more time. Build the missing piece. That's still the whole move.