The SWE-bench leaderboard cannot order its top entries
An audit of 254 published SWE-bench submissions finds that the top systems solve the same issues, that a paired test separates none of the 29 adjacent pairs in the Verified top thirty, and that the scaffold moves a fixed model's score by more than the whole top-thirty spread.
The problem
SWE-bench Verified is a widely quoted test for coding agents. It holds 500 real GitHub issues. An agent writes a patch for each; the official harness applies it and runs the repository's tests. The leaderboard sorts submissions by issues resolved. Teams read that list as an order, select models by it, and claim progress with it.
The authors ask one question: does the published data support that reading? They run no model. They use only what the maintainers publish per submission: a pass or fail verdict on each issue, plus a small metadata file. They audit 254 submissions across the benchmark's four public splits; Verified, with 134 submissions, is the main subject.
What the verdict matrix shows
Think of the published verdicts as a matrix: one row per submission, one column per issue, each cell pass or fail. The authors mine it in three ways.
Most issues no longer separate the leaders. The two leading entries each resolve 396 of 500 issues. Among the top ten, 285 issues are solved by all ten and 51 are solved by none. Those 336 issues add the same amount to every score, so they cannot tell the ten apart. Only 164 issues remain informative. For the top two alone, only 36 issues remain. The authors call this count the effective size of a comparison. It depends on the benchmark and on the systems you compare. Across all 134 submissions the effective size is 94% of the benchmark. Across the top ten it is 33%.
The leaders solve nested sets, not different sets. Two systems with similar scores could solve different issues. The authors measure this with a nesting coefficient: the fraction of the weaker system's successes that the stronger system also achieves. Among the leading systems the median is 0.935. If each system kept its score but its successes were spread at random over the 500 issues, the expected value would be 0.774. So the weaker system's successes are almost a subset of the stronger one's. The union of the top two resolves 414 issues, against 396 for the best single system.
The score belongs to a pair, not to a model. Every submission is a model plus a scaffold. The scaffold is the agent loop, the tool set, the retrieval, and the control policy that drives the model. The leaderboard shows neither as a separate field; only 61 of the 134 Verified submissions carry a usable model tag, so the authors identified each pair by hand.
With the model fixed, the scaffold moves the score a lot. The median range of one model's scores across scaffolds is 78 issues, or 15.6 percentage points. One model appears with nine scaffolds; its scores span 168 to 317 issues, a range of 29.8 points. For comparison, the whole top thirty of the leaderboard spans 8.8 points. Resubmissions of the same scaffold and model on different dates move the score by a median of 27 issues. The scaffold range is 2.9 times that resubmission range.
The two factors also interact. The authors find nine cases where two models and two scaffolds were all submitted in every pairing. Six of the nine interactions stay significant after a correction for running nine tests. In two cases the order of two scaffolds reverses when the model changes. So you cannot subtract a fixed scaffold effect and recover a model ranking.
Can the ranks be read at all?
Every submission is graded on the same 500 issues, so the correct comparison of two entries is a paired test. The authors use an exact McNemar test, which looks only at the issues where the two systems disagree and asks whether the split could be chance. On Verified, none of the 29 adjacent pairs in the top thirty is separable at the usual 5% threshold. The median gap between neighbours is one issue. The same holds on the Lite split.
A counter-test shows the tool is not the problem. The full Test split has 2,294 issues, and its top 24 entries span 0.2% to 52.6%; there the same test separates 14 of 23 adjacent pairs. The fault is that the leaders on Verified have converged on the same issues, not the benchmark family.
The authors then group the top thirty into tiers. Go down the sorted list; an entry joins the current tier if the test cannot separate it from the tier leader, otherwise it starts a new tier. This gives three tiers of 8, 12, and 10 systems, covering 382 to 396, 362 to 378, and 352 to 359 resolved issues. With a stricter multiple-test correction it gives two tiers of 19 and 11. A tier does not mean the members are equal. It means the data cannot order them.
What does not fix it, and what does
The obvious remedy is to retire the issues that every leader solves. The authors rescore the top thirty on only the 209 issues that are informative for the top twenty. The visible spread more than doubles, from 8.8 to 18.7 points, and three adjacent pairs swap places. But separable pairs stay at zero of 29, because the paired test already ignores issues both systems agree on. Retirement saves evaluation cost; it does not add separating power.
Adding more issues of the same kind mostly adds more that all the leaders pass or all fail. To separate the median leading pair at its observed disagreement rate, the benchmark would need roughly 52 times as many issues, about 26,000. Issues that split the two systems, and lean toward one of them, change that. The authors measure the lean as an imbalance rate; at 0.2, about 900 issues are enough. That is their curation test for a new issue: does it break the nesting?
The authors release the analysis as a five-step audit tool: effective size, nesting coefficient, paired test on every pair, tiers with a stated rule, and the count of new issues a comparison needs. Every step reads only the published verdicts.
What you can do with it
For teams that build an internal benchmark to choose a model, the authors give a direct list.
Test the deployed pair. Scaffold orderings reverse across models, so test your model inside your own harness, not with an external ranking. Size the suite for the decision: with 100 tasks, the smallest gap you can detect between independent runs is roughly 17 points, so state the gap you need before you test. Track the effective size for your candidates. Tasks that all candidates pass or fail do not distinguish them. Grade from stored artefacts, never from the agent's own report. Persist the patch and the grader's output per task, and reject patches that touch test files. Decide before the run whether a timed-out task counts as a failure or is excluded, and publish the count of missing verdicts beside the score. Report tiers with cost and latency per tier, not a ranked table. Prefer tasks from your private repositories, and refresh them, because an internal suite fills up with tasks every candidate solves just as a public one does.
The authors state the limits. The audit covers one benchmark family and one run per submission, so run-to-run variance is not separated out. The scaffold effects are associations, not causal estimates, because teams choose scaffold and model together; a controlled study is future work.