GitHub Copilot finished one coding task faster, once

One task, timed, under controlled conditions

On 13 February 2023, Microsoft Research economists posted a working paper reporting a controlled experiment with 95 professional developers, all familiar with JavaScript, randomly split into two groups. Both were asked to build the same HTTP server from the same instructions; one group had access to GitHub Copilot and the other did not, and submissions were automatically scored against a test suite. GitHub's own account of the study, first published in September 2022 ahead of the formal paper, describes the same design and adds that submissions were graded blind to group assignment.

What the task was chosen to show, and the result

The Copilot group finished in an average of 1 hour 11 minutes, against 2 hours 41 minutes for the control group, a 55.8% difference, and also completed the task successfully more often (78% versus 70%). The task was deliberately chosen to be well-specified and self-contained, the kind of boilerplate-heavy work Copilot's training makes it good at completing. GitHub separately reports a survey in which most respondents said Copilot helped them stay in flow and conserve mental effort on repetitive work; that is a self-reported sentiment measure, not part of the timed result, and the two should not be quoted as though they were a single figure.

Why one task time is a data point, not a rate

A 55.8% difference on one task, with one tool, on one kind of well-specified coding problem, describes that comparison and nothing beyond it. It does not describe how Copilot affects a developer's average week, a large or unfamiliar codebase, or maintenance work rather than fresh boilerplate. A later randomised trial by METR, run on sixteen experienced developers working in codebases they already knew, found AI assistance added 19% to completion time rather than cutting it, using different tools in a different setting two years on. Neither result invalidates the other; they measure different tasks, different developers and different tools, which is exactly the point: a single controlled task time is evidence about that task, not a general productivity multiplier waiting to be applied elsewhere.

Questions to ask before you adopt it

• Is the task you care about well-specified boilerplate, or open-ended work in a codebase the model has not effectively seen?
• Is the comparison a timed measurement, a self-reported survey, or both quoted together as if they were one number?
• Would a different study population, such as more experienced developers on familiar code, be likely to show the same direction of effect?

Controlled timing experiments are valuable precisely because they isolate one comparison cleanly. The discipline required is treating that comparison as what it is: a measurement of one task, not a forecast for every task a tool might touch.