Why AI Coding Productivity Changes With Developer Experience and Task Design

Claims about AI coding speed often compress different outcomes into one phrase. A study may count completed tasks, time per task, lines of code, or later maintenance. Those measures answer different questions. A faster first draft does not automatically mean a correct feature, maintainable code, or stronger skill. The evidence is easier to interpret when the outcome stays visible.

You don't have to scroll every AI thread, track every new tool, or watch every demo. 

The Rundown AI breaks it all down for you — the latest AI news, tools, and tutorials in one free 5-minute email every morning. 

Trusted by 2M+ professionals at Apple, Google, and NASA.

Sign up to start learning.

A 2026 preprint meta-analysis examined 23 studies with 27 effect-size estimates for productivity and learning. Its productivity synthesis included 14 studies, with participant and repository-level data. The authors found that coding assistants can raise measured productivity on average, but results varied strongly by context. Their review found mixed learning outcomes. It also searched studies published only through 2025.

In three randomized field experiments, researchers affiliated with Microsoft analyzed data from 4,867 developers at Microsoft, Accenture, and an unnamed Fortune 100 firm. They reported 26.08% more completed tasks among developers with access to an AI assistant. The authors also reported a standard error of 10.3%. This study measured work inside participating firms, under their tools and task definitions.

A different randomized study by METR involved 16 experienced open-source developers completing 246 tasks in mature repositories they knew. It found that tasks took 19% longer on average when early-2025 AI tools were allowed. Afterward, developers estimated a speed gain despite the measured delay. The authors cautioned that results describe a small sample and a specific period of tool capability.

Top engineers at Anthropic say AI now writes 100% of their code.

Are you using AI to write yours?

These 100+ Claude Code hacks show you exactly how. Sign up for The Code and get:

Claim your free playbook

A new 2026 journal study tested automatic versus on-demand code suggestions with 33 developers, split between 16 experts and 17 novices. Participants handled two Java maintenance tasks in a controlled setting. The paper reported higher frustration and lower productivity for experts using automatic suggestions, while novices showed lower frustration and higher measured productivity. It did not test every development workflow.

These results do not cancel one another. The Microsoft experiments counted completed tasks in company settings. METR measured time on real issues in developers’ own projects. The 2026 study compared how suggestions appeared during narrow maintenance tasks. Differences in experience, task, tool, and outcome can change what a study detects. None establishes a universal speed effect for every developer.

In sensitivity checks, I-squared remained between 98.8 and 99.6 percent. The meta-analysis helps place individual findings in context, but it also reports high between-study variation. It combines studies that differ in tool versions, programming tasks, participants, and research settings. Its search ends before 2026 results, so it cannot include the November journal paper. As a preprint, its pooled estimate should be read as a synthesis with limits.

Measurement also changes how employers and teams interpret AI use. Lines of generated code may rise without showing whether a task was completed correctly. Shorter completion time may hide later review, correction, or maintenance work. A reliable assessment separates output, time, quality, and learning instead of turning them into a single productivity score.

For an individual workflow, the evidence supports testing the actual task and reviewing the resulting code. That is an inference from the differences between the studies, not a guarantee that a particular setup will improve work. Teams should state what they count before comparing results, and keep the test conditions alongside the number.

The current research points to conditional effects, not a fixed multiplier. A regular check-in cannot predict each team’s result, but the task and measurement method belong beside any claim about speed.

Are you ready to reach holiday shoppers on CTV? For many streamers, the season is already underway.

Last year, according to Roku internal data, holiday content searches on Roku saw an 86% increase during October. And in November, those searches rise an average of 39% weekly. This surging holiday spirit is mirrored by consumer activity, as about half of Roku streamers shopped Black Friday and Cyber Monday deals in 2025.

Meeting those shoppers is easier than you might think. Begin with the seasonal outcome you hope to drive: awareness, online sales, leads, store visits, or another measurable goal. From there, define your audience, upload your creative, set the budget and campaign dates, and establish measurement before launching. You can even create a shoppable ad unit that lets streamers buy with a click of their remote.

Our holiday playbook offers a month-by-month guide to reaching shoppers during this all-important time of year.

First-time spenders get a $5,000 match on the first $5,000 spent with code BEEHIIV5K. Terms apply.

Launch your campaign today.