Rows of small brass balance scales on a long workbench, each weighing a sealed envelope against an open notebook. Most tip gently toward the notebook, two tip hard, a few hang perfectly level, and an inspector with a lamp walks along the rows. Editorial illustration in a risograph print style: confident ink linework, two muted inks (deep slate blue and warm ochre) on an off-white paper ground, visible grain, generous negative space. No text, no letters, no numbers, no logos.
If you run coding , you have probably wondered whether the extra machinery is worth it. Is a debate between two models better than simply asking your best model? We published two single cases (the first and the second). This post puts every case we ran on one table: four plans, 21 questions, with the strong model's answers written down before the debate began.
Update, October 17: we have since scored eight more plans, and run the equal-budget comparison this post said was missing. The original count below is unchanged. The new results are in the update at the end, and they are less one-sided.
How we ran it
Each of our plans starts with a prep pass: an agent reads the code and writes the plan's open questions, each with its options and what hangs on them. For these four plans, the leading our work then answered every question on its own and saved those answers to a file before any debate started. That is : the solo answers were fixed before anyone could see what the debate would decide.
Then a debate walk decided the same questions from the same prep. A walk is not an open-ended argument: one model presents each question and its options, another rules on it with the code open, and each ruling is recorded against lettered options with its reasons. On one plan we ran the walk twice on byte-identical preps, once with two frontier models and once with two local . Finally we compared the solo answers with the debate's decisions, one question at a time.
The count
| Plan | Questions | Debate overturned the solo answer | Same direction, debate improved it | Matched | Values split |
|---|---|---|---|---|---|
| How a walk reports its rulings | 4 | 0 | 4 | 0 | 0 |
| Landing accepted work automatically | 6 | 2 | 3 | 0 | 1 |
| A judge for walk rulings | 6 | 0 | 3 | 3 | 0 |
| A memory leak in a web service | 5 | 0 | 4 | 1 | 0 |
| Total | 21 | 2 | 14 | 4 | 1 |
On none of the 21 questions did the debate's decision come out worse than the solo answer. The debate also raised two questions the solo model never considered. It made one mistake of its own: a contradiction between two of its decisions, which an independent read of the whole plan caught before anything was built.
The two overturns
Both came on the plan for landing accepted work automatically, and both pairs of debaters made them independently. The solo model had kept a person as the trigger for each landing, which is the very hand-work the change existed to remove. It had also answered the question of how to combine commits by declining its premise, which made a later question “dissolve”: it had answered a different plan than the one filed. The frontier pair and the local pair, debating the same prep separately, overturned the same two answers, and on review both overturns were right. Two different model pairs making the same corrections is the strongest sign in this data that the effect comes from the process rather than from one pair's ability.
What “improved” looks like
Most of the gain is depth in the same direction. A few examples:
- A completeness check. The solo answer accepted a report of rulings; the debate added that the number of rulings must equal the plan's own count of questions, so a run that crashed halfway can't be ingested as if it had finished.
- Work that survives a failure. Where the solo answer rejected a whole report over one bad ruling, the debate invented an option outside the presented set: keep every ruling before the bad one and re-run only from there.
- A governance catch. The solo answer let a local model re-rule decisions the person had already recorded. The debate saw that a model must never be in a position to veto the person's own decision, and replaced the re-ruling with a check that calls no model at all.
- One read path, not two. On the memory leak, the solo fix added a second, lean query beside the full read. The debate fixed the shared read at its source instead, so all six callers got cheaper and there was one way to read the records rather than two that could drift apart.
The values split was a real choice, not a mistake on either side: whether a person must still attest to a documentation audit. The frontier pair removed the person, while the solo answer and the local pair kept them. It went to a person to decide.
What this doesn't show
- The effort wasn't matched. A debate spends more than one model answering once. Controlled studies find that much of what multi-agent setups gain comes from that extra effort (Bertalanič and Fortuna, 2026). The comparison we haven't run is a solo arm given the same budget, such as two independent solo plans reconciled. That test is on our queue, and we'll publish what it finds. (We have now run it: see the update.)
- The judge was the solo author. The model that wrote the solo answers also compared them with the debate's. If anything that tilts the count toward the solo answers, since models tend to prefer their own output (Pombal et al., 2026), but it is still one judge's reading.
- Four plans from one system. All four are design questions in one codebase. They are real work rather than a , which is the point, and also the limit.
What we take from it
A strong model answering alone gets the direction right most of the time: it matched or shared the debate's direction on 18 of 21 questions. What it misses are the things off its own path: a check that makes a failure visible, an option nobody listed, a person's decision a model shouldn't touch, a measurement in place of an assumption. A second reader working from the same prep, with the code open and every ruling written down, finds those. Our walks are built around that, and so is the single independent reader that follows them, which in a separate sample found real design gaps in 17 of 30 of its findings.
Update: eight more plans, and the equal-budget test
Since this post went up, we have scored a second batch: eight more plans, 36 questions. This batch was built to answer two of the limits above.
- The solo side was isolated. Each solo answer came from a fresh frontier model in a that could see the code and the plan's prepped questions, and nothing else: no plan file, no earlier decisions, no write-ups. Every plan was prepped but not yet debated when its solo runs started, so no answer existed anywhere to leak.
- The solo side got a second budget. Two solo runs answered each plan independently, and a third run reconciled the two into one set of decisions. That reconciled answer is the equal-budget arm: three model runs against one debate walk.
All solo answers were saved before anyone read the walk's decisions. The walks themselves were ordinary ones, with the pairing of models varying from night to night.
| Plan | Questions | Debate overturned the solo answer | Matched | Values split | Debate went wider | Solo did better |
|---|---|---|---|---|---|---|
| Landing ready work in batches | 7 | 2 | 3 | 2 | 0 | 0 |
| A guard against editing shipped migrations | 6 | 0 | 5 | 0 | 1 | 0 |
| A corrected work record that its own links locked out | 4 | 0 | 3 | 0 | 0 | 1 |
| Documentation drift eating the review budget | 5 | 2 | 2 | 1 | 0 | 0 |
| Agent commits that skip the | 4 | 3 | 0 | 1 | 0 | 0 |
| Repairing a migration edited after it shipped | 3 | 1 | 2 | 0 | 0 | 0 |
| A test that failed only some of the time | 4 | 0 | 4 | 0 | 0 | 0 |
| A security filter that blanked a service's key | 3 | 0 | 2 | 1 | 0 | 0 |
| One solo run, total | 36 | 8 | 21 | 5 | 1 | 1 |
| Three solo runs reconciled, total | 36 | 8 | 20 | 4 | 3 | 1 |
This batch was scored more strictly than the first. Where the two sides reached the same decision and the debate only added detail, it counts as matched; the first table would have counted many of those as “improved”. So the two tables don't add up into one, but the overturn counts are comparable: 2 of 21 in the first batch, 8 of 36 in this one.
What the overturns were
Half of the eight share one shape. The solo model proposed to build something new (a check, a field, a record), and the debate, reading the code, found that the system already did it. In one plan the solo answer added a new way to record test runs for data the store already kept; in another, it added a check for a bypass that two existing checks already refuse. The solo model reasoned from the problem as described; the debate checked the description against the code.
The other four were judgment calls where the scoring found the debate's choice better. The clearest: asked how to fix a database migration that had been edited after it shipped, both solo runs chose to edit the shipped migration again, which is the very mistake the plan existed to stop. The debate left the shipped version alone and put the fix in a new one.
Where the solo model did better
“None worse” does not survive the second batch. Twice, the solo model found something the debate did not.
- It answered a question the debate deferred. Asked what had released a stuck piece of work, the solo model read the branch history and found the exact commit and time. The debate ruled the release unexplained and left it for later. Its caution was defensible, but the solo answer was right, and we confirmed it.
- It found the real bug. On the test that failed only some of the time, we had written our own guess at the cause on the plan before it was prepped, so every side started from that guess. The scores show four matches. But only the solo model looked past the guess: it flagged that the code recognising commit ids skipped ids made only of digits, which is what was actually failing the test, about once in every hundred runs. The debate decided all its questions around our wrong guess. The fix, when it was built, confirmed the solo answer.
What the extra solo budget bought
Very little. Reconciling two independent solo answers changed the score on 3 of 36 questions. On one it closed the gap and matched the debate. On the other two it made the answer narrower, dropping something that both the first solo run and the debate had kept. All eight overturns stood: when the first solo run missed what the code already did, the second run missed it too, and reconciling two answers that share a blind spot does not remove it.
That is one result on one codebase, and it doesn't settle the research question. But it is the comparison the studies above say matters, and here, more of the same solo effort did not buy what the debate bought.
What the update doesn't show
- The budgets were sized, not metered. We sized three solo runs to roughly one walk's time, but didn't measure either side's . Each solo run took three to six minutes and cost a dollar or less; these walks ran about 20 to 40 minutes. If the budgets are off, they are off in the debate's favour.
- The judge was the same kind of model. The scoring was done by the frontier model leading our work: the same model family as the solo runs, though not the same runs.
- Still one codebase. All eight plans are design questions in the same system as the first four.
- One plan carried our own framing. On the test that failed some of the time, our guess at the cause was in every side's brief. We kept the plan in the count and say so here.
- Nothing checked a decision against an outcome. Both batches score one set of decisions against another; neither side's plan was built and tested. Work on agent points to how that could be measured: Google Research's Harness² improves a fixed model by changing only its harness, and it names a coding benchmark whose tests the agent never sees. Debate and solo plans, each implemented at equal effort and scored by tests like those, would give this count the ground truth it lacks.
What we take from both batches together: a strong model answering alone gets the direction right most of the time, and spending more on it doesn't change much. The debate's distinctive catch is checking the premises against the code. But the debate can also be led by a wrong premise, as it was when we handed it one, and a single solo reader then saw what it didn't. Ten more plans have their solo answers saved and are waiting to be scored, and we'll add them here.