Evidence note

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Controlled Experiment2025METR

What was examined?

A randomised study of experienced open-source developers completing real issues in mature repositories they already knew well.

Tasks were randomised between AI-allowed and AI-disallowed conditions.

16 experienced developers; 246 real repository issues.

Key findings

1Developers took approximately 19% longer when AI tools were available in the studied setting.
2Before the study, participants expected AI to make them substantially faster.
3After the study, participants still believed AI had improved their speed despite the measured slowdown.

Why it matters. Lornets interpretation.

Perceived productivity and measured productivity can diverge. AI productivity should be evaluated in the context of the actual developer, repository, task and tool rather than assumed from general adoption.

This is the Lornets reading of the source, not a finding of the source itself.

What it does not establish

1The sample was small.
2Participants were unusually experienced with the repositories.
3The study reflects early-2025 AI tooling.
4METR explicitly cautions against generalising the result to software development broadly.