Evidence tension

When does AI actually make software development faster?

AI-assisted development is often discussed as though it has one universal productivity effect. Current evidence indicates that the measured effect depends on the developer, task, repository, tool and outcome being measured.

Evidence confidence: High

What the evidence shows

Substantial gains in large field experiments
Three randomised field experiments covering 4,867 professional developers found a positive combined effect on completed development tasks.
EVD-0011
A measured slowdown in experienced maintainers
A controlled study of experienced open-source developers working on real issues in repositories they knew well measured slower task completion when early-2025 AI tools were available.
EVD-0013
Local development gains meet system constraints
DORA's research suggests AI acts as an amplifier of the wider engineering environment and that increased development velocity can interact with downstream delivery constraints.
EVD-0009

The cited studies measure different outcomes and should not be presented as directly comparable effect sizes.

How the evidence compares

What can be compared

The evidence can be used to reject the idea that one universal AI productivity effect applies across all software-development settings.

What cannot be directly compared

The numerical effect sizes should not be placed on one common scale as though completed tasks, repository task duration, delivery throughput and perceived productivity are the same outcome.

Reasons for the difference

Developer experience
Repository familiarity
Task characteristics
AI tool generation
Outcome measurement
Downstream delivery constraints

Lornets interpretation

AI productivity is conditional rather than universal. Code-generation speed, developer task productivity, delivery throughput and production outcomes should not be treated as interchangeable measures.

Remaining uncertainty

AI tools are changing faster than long-running empirical research can fully capture them. Existing studies therefore provide evidence about particular generations of tools and development contexts rather than a permanent productivity coefficient.

Evidence needed

Larger controlled studies using newer tools across different engineering contexts, with outcomes followed from developer activity through delivery and production performance.