Devin's 2024 launch paired a demo with a vendor-run benchmark

An announcement dated 12 March 2024

On 12 March 2024, Cognition announced Devin, described in its own post as 'the first AI software engineer', able to 'plan and execute complex engineering tasks', 'learn how to use unfamiliar technologies' and 'contribute to mature production repositories'. The centrepiece supporting evidence was a benchmark figure: a reported 13.86 per cent success rate on SWE-bench, which the post says was 'far exceeding the previous state-of-the-art of 1.96 per cent'. Both the demonstration and the number are the company's own account of its own product, published on the day it wanted attention for a launch; neither is an independent measurement.

What SWE-bench actually measures

Cognition's separate technical report, dated 15 March 2024, explains the benchmark rather than just citing the score. SWE-bench draws real GitHub issues and pull requests from popular open-source Python repositories, and scores a system on whether its generated patch makes the project's own tests pass. Cognition says it evaluated Devin on 'a randomly chosen 25 per cent of the SWE-bench test set', resolving 79 of 570 sampled issues, and reports that even the strongest prior assisted approach only reached 4.80 per cent when given the exact files to change. That is the comparison the 13.86 per cent figure rests on: a self-selected sample, scored and reported by the company that built the system.

What the report does not settle, and where review sits

The same report names its own limits. It acknowledges that Devin's underlying models may already contain material from the repositories being tested, a form of data contamination that affects competitors too but is not eliminated. It also notes that an agent with internet access could look up a fix externally rather than deriving it, and says runs were manually inspected to guard against that rather than ruled out by design. None of this makes the demonstration false; it means the number is a vendor-run result on a partial sample, not a replication by an outside lab. On the workflow itself, Devin's own documentation describes the system producing 'draft PRs waiting for review' and lets a person 'follow and take over Devin's development process' as it runs — human review sits after Devin proposes a change, at the pull request, not before it acts.

• Is a headline benchmark figure independently reproduced, or only reported by the vendor that built the system?
• What fraction of a benchmark's test set was actually evaluated, and how was that subset chosen?
• At what point does a person see and approve a change: before it runs, or only once a pull request is already open?

A demo and a benchmark score both show that something worked in the cases shown. Neither tells a reader how the system performs on their own repository, under their own review process, once the vendor is no longer choosing which runs to publish.