Free models agree with the expensive verifier when it says yes, but miss most of its real failures

In a two-agent dispatch contract, an orchestrator hands one unit of work to a worker: a single card naming the task, the files to change, and a set of numbered acceptance criteria. The worker is the agent that writes the code, inside a sandbox that limits what it can touch. The verifier is a second agent, run outside that sandbox, that grades the finished work against the same numbered criteria and returns a PASS or FAIL verdict. Nothing merges without a PASS.

The verifier is the expensive half of that arrangement, because it is usually a frontier model: the strongest commercially available tier, billed per token. The obvious question is whether something free could do the job instead. This study tested eight candidates: five open-weight models served locally through Ollama (gpt-oss:120b, qwen3-coder:30b, qwen3:32b, devstral, and llama4:scout) and cloud free tiers from Groq and OpenRouter, the latter fielding two Nemotron models after its free roster rotated away from the ones originally planned, one day after they were chosen.

That roster churn was the first finding before any model was scored: a free-tier model id that exists today may not exist tomorrow, so anything that hardcodes one needs a liveness check at run time, not at authoring time.

The benchmark is ten real stages of agent-written work, each already carrying a frontier verdict recorded during normal operation: a genuine PASS or FAIL judgement per numbered criterion, produced when the work was originally reviewed. Six of the ten contain at least one criterion the frontier failed. Each free candidate was shown the same stage card and the same deliverable files, in a single request, and asked to produce its own verdict for the same numbered criteria. Because both sides grade the same numbered list, criterion 3 in one verdict is criterion 3 in the other by construction, with no fuzzy matching.

Three scores come out of that. FAIL recall is the fraction of the frontier's failures the candidate also caught, and it is the number that matters: a verifier that misses a real failure lets broken work merge. PASS agreement is the fraction of the frontier's passes the candidate also passed. Artefact discipline is the fraction of attempts that produced a schema-valid verdict at all; a model that returns malformed JSON scores zero for that stage regardless of what it meant to say.

Every candidate ran single-shot, local models included, so the comparison measures judgement rather than retrieval: the same packaged evidence went to every model, and none could go and re-read a file the others could not.

The headline pattern is consistent: PASS agreement runs high across the board, between 72% and 98%, and it is a deeply misleading number. Most criteria in most stages pass, so a model that says yes to everything scores well on agreement while protecting nothing. Llama 4 Scout is the cleanest example: the highest PASS agreement measured, 98%, alongside a FAIL recall of exactly zero. It did not independently catch a single one of the eleven frontier-failed criteria it was shown. That is not caution; it is a verifier that says yes by default.

Only two candidates cleared a defensible bar: gpt-oss:120b running locally and OpenRouter's nemotron-3-ultra-550b, tied at 77% FAIL recall. Even that best case misses roughly one real failure in four. Groq's free tier never got to compete: its 8,000 tokens-per-minute ceiling is judged against the prompt plus the requested output budget, and the verifier prompt alone exceeds it on most stages, so the same weights that scored 77% locally managed one usable attempt out of ten through Groq. A capacity verdict, not a quality one.

Artefact discipline had its own casualties. Devstral twice returned JSON in the wrong shape, and Llama 4 Scout produced the study's strangest failure: asked to fill in a verdict conforming to a schema, it returned the schema document itself, blank, after 38 minutes of computation. Both models landed at 80% discipline, the joint worst among candidates that ran.

Local inference is free but not fast, and the cost is not where intuition puts it. For the largest candidate, the 67GB llama4:scout, elapsed time tracked the size of the prompt far more closely than the size of the answer: output stayed in a narrow 800 to 3,500 token band while wall clock swung from 14 to 38 minutes with the amount of evidence being read in. Prompt ingestion, not generation, is the binding cost.

That showed up as a timeout ladder. At the 420-second default every one of Scout's ten stages timed out. At 900 seconds, only the stage with the smallest prompt finished. It took 2,400 seconds, 40 minutes per verdict, to get eight of ten through, and the two that still failed did not run out of time; they returned unusable output well inside the window. The practical lesson generalises: a timeout for a large local model should be sized from prompt tokens, not set as a flat per-model constant.

Speed also cuts the other way. The fastest candidate, qwen3-coder:30b at 46 seconds, caught only 15% of real failures. A fast, cheap, confidently wrong verifier is precisely what an unattended overnight loop would learn to trust, which makes its speed a hazard rather than a feature.

The recommendation is blunt: none of the eight can be trusted as the verifier without qualification, because even the best misses roughly a quarter of real failures, and that false-negative rate is intolerable for a gate deciding whether broken work merges. What the best two can do is mechanical screening: run gpt-oss:120b locally as a first pass, and send anything borderline, and anything it fails, up to the frontier. Cheap re-queues are an acceptable cost; silent merges of broken work are not, and the rubber-stamp candidates buy exactly the wrong one.

The local route wins the tie-break with the cloud free tier on everything except speed: no daily request cap, no rotating model roster, and no third-party exposure, since every cloud verification ships the stage card and code diff to someone else's API. For a repository whose diffs must stay on the machine, local weights are the only free option at all. Even a proposed one-off $10 payment that would raise OpenRouter's daily request cap fails the maths: the free local candidate already matches its failure recall at zero cost.

The wider finding is worth keeping even if none of the model names survive the year. Judging finished work turns out to be a harder capability than agreement statistics suggest, and any evaluation of a cheap judge has to score what it catches, not how often it concurs. Concurrence is the one thing a yes-machine does perfectly.