Evidence note

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

Controlled Experiment2025METR

What was examined?

A randomised study of experienced open-source developers completing real issues in mature repositories they already knew well.

Tasks were randomised between AI-allowed and AI-disallowed conditions.

16 experienced developers; 246 real repository issues.

Key findings

01Developers took approximately 19% longer when AI tools were available in the studied setting.
02Before the study, participants expected AI to make them substantially faster.
03After the study, participants still believed AI had improved their speed despite the measured slowdown.

Why it matters. Lornets interpretation.

Perceived productivity and measured productivity can diverge. AI productivity should be evaluated in the context of the actual developer, repository, task and tool rather than assumed from general adoption.

This is the Lornets reading of the source, not a finding of the source itself.

What it does not establish

01The sample was small.
02Participants were unusually experienced with the repositories.
03The study reflects early-2025 AI tooling.
04METR explicitly cautions against generalising the result to software development broadly.