Loading…
Glossary term
A benchmark that tests whether models can resolve real issues from software repositories.
SWE-bench gives a model a real bug report or feature request from a popular open-source Python project, along with the codebase. The model must produce a patch, which is checked by running the project's tests.
Variants such as 'Verified' use a human-checked subset of tasks.
It tests realistic software engineering rather than toy puzzles, and is widely used to compare coding agents.