Testing Multi-Agent Coding Teams Reveals a Narrow Quality Gain Beside Higher Inference Costs

Vals AI published a case study on October 9. It tests whether dividing a coding task among agents improves results enough to justify added inference. The evaluation used 50 full-stack web-app assignments. GPT-6 Sol and Claude Opus 5.5 each worked alone and in lead-plus-subagent teams, at medium and maximum reasoning effort.

Turns out, AI is good for more than writing emails and generating LinkedIn posts. This guide is packed with 200+ actionable ways to build income with AI.

Inside you'll discover:

AI is changing how people work and how people earn, far beyond the simple “write me an email” prompt. Discover the possibilities and download the free guide to start cashing in today.

Get The Free Guide

Across four matched comparisons, teams cost between 1.8 and 5.1 times as much as single agents. Only one team result showed a statistically significant score gain. GPT-6 Sol's medium-effort team scored 7.3 points higher than its single-agent version. The other differences ranged from a small decline to a modest increase, but were not statistically significant in this test.

The numbers do not show that teams are simply worse. Sol's medium-effort team scored 84.9 percent against 77.6 percent for one agent. At maximum effort, the team scored 90.4 percent against 89 percent for one agent. Claude Opus 5.5's maximum-effort team reached 93.2 percent, but its 3.4-point lead was not statistically significant. Its average cost per application was 122 US dollars, against 23 dollars and 77 cents for one agent.

Vals built its benchmark around complete web applications. Each system received a product specification, deployed an app, and faced browser-based interface tests. The researchers paired results across the same 50 applications and used statistical tests to compare outcomes. This measures passed tests on generated apps. It does not directly measure maintainability, security, code review quality, or success on every software task.

The team's internal work also helps explain its cost. A lead agent split the app into parts and assigned them to subagents. Each subagent maintained context about the specification and its work. Vals reports that the extra spending came mostly from cached input tokens, because this context was sent again with model calls. Coordination and repeated context can therefore consume resources even when the visible final answer appears to be one application.

You don't have to scroll every AI thread, track every new tool, or watch every demo. 

The Rundown AI breaks it all down for you — the latest AI news, tools, and tutorials in one free 5-minute email every morning. 

Trusted by 2M+ professionals at Apple, Google, and NASA.

Sign up to start learning.

Time did not move in the same direction for both models. Sol's teams finished somewhat faster than its single agents, especially at maximum effort. Opus's teams took longer, reaching more than twice the single-agent runtime in one configuration. The evaluation therefore presents a trade-off among quality, cost, and completion time. Parallel work can reduce elapsed time in some cases without improving the test score significantly.

A separate preprint examined a different multi-agent setup under a fixed model-call budget. It compared a planner, executor, and critic arrangement with a single optimized agent on ALFWorld and WebShop tasks. The authors found no statistically significant team advantage in their reported comparisons. This is related evidence, not a replication of Vals: it used a frozen seven-billion-parameter model and different tasks.

The boundaries matter. Vals examined two models, two reasoning settings, and web-app construction. The result does not establish that agent teams are wasteful for research, parallel analysis, or tasks with independent subtasks. Nor does it show that a single agent is always faster. It identifies conditions where adding agents raised measured costs more consistently than it raised benchmark scores.

The research value is its cost-aware comparison. A quality score by itself can hide the extra model calls behind a team result. Here, three of four comparisons showed no statistically significant score gain, while every team cost more. Wider task coverage and comparable reporting of quality, cost, and time are needed before treating this as a general rule. This is one research check-in, not a universal rule; measurements from each real task should guide decisions about when teams fit.

Supercharge your productivity with HubSpot's comprehensive guide to Chat GPT. This free resource is your fast track to AI mastery:

Plus, in-depth sections on email composition, content creation, customer support, and data analysis.

Get The Free Guide