When SWE-bench Saturates: Reading Agent Benchmarks in 2026

In July 2026, an open-weight model resolved 93.4% of SWE-bench Verified — and the number came from a referee, not the vendor. When the original SWE-bench debuted in late 2023, the best published result was under 2%. That is the whole arc of agent benchmarking in one sentence: the field’s favorite yardstick went from impossible to nearly solved in under three years, and somewhere along the way the scores quietly stopped telling you which model to use.

This piece is about reading agent benchmarks now that the headline one is saturating: the mid-2026 numbers and what they actually say, the two gaps that explain most score confusion, why a 90% model can still fumble your Tuesday, the successor suites, the vendor dress-up moves and the referees who undo them — and the cheap private benchmark that beats all of it. The reading checklist near the end is the part worth bookmarking.

The SWE-bench saturation story, in four numbers

SWE-bench is a benchmark that tests whether an AI system can resolve real GitHub issues: each task hands the agent a repository snapshot and an issue description, and scores success by whether held-out tests pass after the agent’s patch. It was introduced in October 2023 with 2,294 tasks drawn from twelve popular Python repositories (the SWE-bench paper).

The early numbers were humbling — the paper’s best configuration resolved fewer than 2% of tasks. Then the ramp: in August 2024, OpenAI and the benchmark’s authors released SWE-bench Verified, a 500-task subset screened by professional engineers to remove underspecified and unsolvable items, and Verified became the industry’s default coding-agent scoreboard (swebench.com).

By mid-2026, the open-weight column alone reads like a solved problem. Four numbers from the summer scorecard (mid-2026 open-model roundup), all on SWE-bench Verified:

• Kimi K3 (Moonshot AI, July 16, 2026): ~93.4%, as measured by the independent eval firm Vals AI (vals.ai)
• DeepSeek V4 Pro (GA July 19, 2026): ~80.6%, vendor-reported
• GLM-5.2 (Z.ai, June 16, 2026): ~78.7%, as measured by Epoch AI (epoch.ai)
• Qwen3-Coder-Next (Alibaba): ~70.6%, vendor-reported

These are open-weight models — the full cards, licenses, and prices are in the open-weight scorecard — and they are crowding a ceiling the proprietary frontier reached first. When models cluster in the high 80s and 90s, a benchmark stops discriminating: the remaining unsolved tasks skew toward ambiguity, grader quirks, and environment flakiness rather than capability, and a two-point lead means roughly nothing. Statisticians call it a ceiling effect. Vendors call it a press release.

None of this is new, which is exactly why it’s predictable. HumanEval went from frontier-defining to a footnote once pass rates hit the high 90s; MMLU followed the same arc for knowledge. Coding-agent benchmarks are simply the latest instance of the pattern: a bench is born discriminating, gets optimized against, and retires into a regression test. The only question worth asking in August 2026 is where on that curve each number you’re reading sits.

Two gaps that do the real explaining

Saturation is only half the reading problem. The other half is that “the score” was never one number — it’s a number conditioned on which benchmark variant and who ran it, and mid-2026 gives us a clean specimen of each gap.

The Verified-vs-Pro gap. GLM-5.2 scores ~78.7% on SWE-bench Verified per Epoch AI — and ~62.1% on SWE-bench Pro per its own vendor. Same model, same summer, 16.6 points apart. SWE-bench Pro is Scale AI’s harder successor (scale.com): a public set built from copyleft-licensed repositories specifically to resist training-data contamination, plus a held-out commercial set, with longer, multi-file, enterprise-shaped tasks. When frontier models first met Pro in late 2025, scores landed in the low twenties. Nothing about GLM-5.2 changed between those two numbers. The tasks did — they stopped resembling anything the model might have seen or memorized.

The vendor-vs-independent gap. Look at who produced each Verified number above: two came from vendors (DeepSeek, Alibaba), two from referees (Vals AI for K3, Epoch AI for GLM-5.2). Vendor runs typically use in-house harnesses tuned for the benchmark, favorable sampling settings, and publication discretion — you see the run they chose to show you. Referee runs standardize the harness and publish either way. The two kinds of number are not comparable, and mixing them in one comparison table — which every launch-day chart does — is how “our model beats X” claims get manufactured.

Every score ships with two tags — which bench, and who ran it. Strip either tag and the number stops meaning anything.

Mid-2026’s table has a twist worth savoring: the highest number on it is referee-run. That isn’t evidence that vendor spin is over. It’s the saturation signal in its purest form — when even the independent run prints 93, the benchmark has run out of headroom to argue about.

Why a 90% score doesn’t predict your Tuesday

Even a referee-run, ceiling-grade score answers a narrower question than the one you’re asking, for three structural reasons.

Contamination. SWE-bench tasks are public GitHub history, which means they leak into pretraining corpora, and every lab now also trains agents in executable practice environments purpose-built to look like benchmark tasks — the RL environment gold rush exists precisely because agent skills come from practicing agent work (Epoch AI on the state of RL environments). The line between “trained on the test” and “trained on ten thousand things shaped like the test” is real but thin, and only fresh, held-out tasks can tell you which side of it a score sits on. That’s the entire design rationale of Pro’s commercial set.

Harness effects. SWE-bench leaderboards have always listed scaffold-plus-model pairs, not bare models — and the same model swings dramatically across scaffolds. The score you read is a system score: model, agent loop, tools, retry policy, prompts. Unless you run the vendor’s exact harness (you won’t; it’s usually unnamed), their number was measured on an instrument you don’t own. Which harness you drive day to day is its own consequential choice — that’s the agent harness field map — but it means benchmark deltas of a few points are routinely smaller than the harness effect hiding inside them.

Task distribution. Verified is 500 self-contained, bugfix-shaped Python tasks with a reproducing test, drawn from a dozen open-source repos. Your Tuesday is an underspecified feature request in a nine-year-old TypeScript monorepo, a flaky integration suite, a review queue, and a migration nobody wants to own. A model can be genuinely superhuman at the first distribution and merely decent at yours. High scores transfer worse the further your work sits from “well-specified small Python fix” — which is most of the distance.

Add the measurement style: most headline numbers are pass@1 under generous budgets, reported without variance. Daily driving cares about the opposite quantity — consistency across runs (pass^k, the probability all k attempts succeed), cost per solved task, and behavior when the task is ambiguous. Goodhart’s law does the rest: once a measure becomes the target every lab optimizes, it stops measuring what made it useful.

The successor benchmarks — and their own clocks

There is no “next SWE-bench,” because the question splintered. The suites that matter in mid-2026 each probe a different axis of agent work:

Suite	What it actually measures	Axis
SWE-bench Pro (scale.com)	Contamination-resistant, multi-file, enterprise-shaped issue resolution	Harder coding
Terminal-Bench (tbench.ai)	End-to-end tasks in a real terminal: builds, sysadmin, data plumbing	Tool-and-shell fluency
τ-bench	Tool use with a simulated human in the loop; scores consistency across repeated runs	Reliability under interaction
OSWorld	Real GUI tasks across desktop apps and OSes	Computer use
METR time horizons (metr.org)	The length of task an agent completes at 50% reliability — doubling roughly every seven months per METR’s methodology	Long-horizon endurance
SWE-Lancer	Real freelance software jobs, scored in dollars of work completed	Economic value

Two honest caveats. First, every successor starts its own saturation clock the day labs begin training against it; benchmarks are consumables, and the useful lifespan of each generation is shrinking. Second, fragmentation cuts both ways — six axes resist single-number hype, but they also let a vendor cherry-pick the one chart they win. The mid-2026 frontier scorecard is our attempt to read the majors across axes at once, honest unknowns included; re-run whatever comparison you rely on quarterly, because every line of it decays.

Which axis should you personally watch? Match it to where your agents actually fail. If yours die mid-task on hour-long jobs, METR-style horizon numbers matter more than any coding score. If they fumble builds and shell plumbing, Terminal-Bench is your leading indicator. If you’re buying reliability for a customer-facing workflow, τ-bench’s consistency framing — not peak performance — is the number that predicts your support queue. A benchmark is only useful when its failure mode rhymes with yours.

How vendors dress up numbers, and the referees who undress them

The dress-up moves are stable enough to enumerate. You will see all six this year:

1. Best-of-N as the headline. The bold number is pass@8 or “with parallel test-time compute”; pass@1 lives in a footnote, several points lower.
2. The unnamed scaffold. “Internal agent harness” — tuned to the benchmark, unavailable to you, occasionally responsible for more of the score than the model.
3. Subset selection. “On a 300-task subset” — selected how? By whom? A subset score is a different benchmark wearing the same name.
4. Unbounded budgets. No tokens, wall-clock, or retry limits reported. Capability at any cost is not a deployment-relevant quantity.
5. Contamination silence. No statement about training cutoff versus task publication dates. Silence is an answer.
6. Decimal theater. Treating 93.4 versus 93.1 as a ranking when run-to-run variance exceeds the gap. At the ceiling, decimals are noise wearing a medal.

The countervailing force is the referee layer that matured in 2024–2026: Epoch AI’s benchmarking work runs frontier and open models through standardized, documented harnesses and publishes methodology (epoch.ai/benchmarks); Vals AI does independent evaluation across coding and professional domains (vals.ai). Referee numbers usually land below vendor numbers — and when they land above, as with K3, that’s information too. An honest benchmark report, vendor or not, names the harness and version, reports pass@1 with run counts and variance, states cost per solved task, discloses task-date-versus-cutoff, and invites replication. Count how many of those boxes a launch post ticks and you have most of your answer about the rest of it.

The benchmark-reading checklist

Eight questions, in the order that saves the most time. A claim that survives all eight is rare and worth your attention.

1. Which benchmark, exactly? Verified ≠ Pro ≠ Lite ≠ full set ≠ “a subset.” Name and version, or discard.
2. Who ran it? Vendor or independent referee. If vendor-only, treat as provisional until Epoch, Vals, or someone with no stake reproduces it.
3. What harness? Named and available, or unnamed and internal? You’re buying the system, not the model in the abstract.
4. pass@1 or best-of-N? And how many runs behind the number — a single run at the ceiling is a coin flip with a press team.
5. Could the model have seen the tasks? Training cutoff versus task dates, plus whether the bench has a held-out set. No statement = assume contamination.
6. What did it cost? Tokens, dollars, or wall-clock per solved task. A score without a bill is half a score.
7. Does the task mix resemble your work? Language, repo size, ambiguity, horizon. Distance from your distribution discounts the number.
8. Is there an independent number yet — and does the delta alarm you? A big vendor-vs-referee gap is a fact about the vendor, not the referee.

Print it, pin it, apply it to the next launch thread. Unanswered questions count as answers.

Build your own private benchmark

Here is the uncomfortable conclusion the two gaps point at: the only benchmark whose task distribution matches your work is the one you build from your work. It’s also far cheaper than it sounds — a private bench of 10–20 tasks beats a public bench of 2,294 for the decision you actually face, which is “should my team switch models or harnesses this quarter.”

The recipe in brief. Harvest tasks from your own recent history: real bugs you fixed, real features you shipped, real reviews you wrote — each with the repo state at the time, the ask as it was actually phrased (ambiguity included), and an acceptance check. A task card needs nothing fancier than this:

# bench/007-webhook-retry.yaml
repo: payments-service @ 3f2c9a1 # commit before the fix landed
ask: >
  "webhooks silently drop when the queue is full — can we not lose these?"
  # verbatim from the ticket, ambiguity preserved on purpose
accept:
  - tests/webhooks/test_retry.py passes # written after the real fix
  - no changes outside src/webhooks/
budget: 400k tokens, 30 min

Twenty of those is an afternoon of curation, not a quarter of tooling work. Hold them out of every agent’s context from now on. Grade on outcome and trajectory — did it pass, and did it get there without touching what it shouldn’t — which is the same discipline agentic QA applies to shipping code. Track pass rate, cost per solved task, and turns per task. Re-run the suite on every candidate model or harness, and re-run it quarterly on your incumbent, because providers change things under you. The full build — task formats, graders, the pass^k math — is in our evals for AI agents guide, and it’s the same daily-driver standard we hold the best agentic AI tools to when we review them: tested on real work, not leaderboard positions.

The step people stall on is the first one — “collect real tasks” — because the raw material is scattered across months of terminal sessions in five different tools. It shouldn’t be archaeology.

Product note: Your archive is your benchmark. Automater Lite consolidates sessions from 10+ CLIs — Claude Code, Codex, Qwen Code, Kimi, OpenCode, Copilot, and friends — into one local, full-text-searchable library, so last quarter’s real fixes become this quarter’s eval tasks, and Vault redaction scrubs secrets before anything turns into a shared test set. Free, on automater.ai.

When the next launch thread fills with bar charts, you’ll have the two tags to check, the eight questions to ask, and twenty tasks of your own that no lab has ever trained on. That last part is the only benchmark advantage left that money can’t buy.

FAQ: SWE-bench and agent benchmarks in 2026

What is SWE-bench Verified?

SWE-bench Verified is a 500-task subset of SWE-bench, released in August 2024 by OpenAI with the benchmark’s authors. Professional engineers screened out underspecified and unsolvable tasks, making it the standard scoreboard for coding agents. Agents get a real repository and issue; success means the held-out tests pass.

Is SWE-bench still relevant in 2026?

As a floor, yes; as a differentiator, barely. With open models scoring 70–93% on Verified, top-end gaps sit inside harness effects and run variance. It still catches regressions and screens weak models, but switching decisions now need harder suites — SWE-bench Pro, terminal and long-horizon benches — plus your own private eval.

What is the difference between SWE-bench Verified and SWE-bench Pro?

Verified is a human-screened 500-task subset of the original public Python set — clean but heavily trained-against. Pro, from Scale AI, uses copyleft and held-out commercial repositories with longer, multi-file, enterprise-shaped tasks to resist contamination. The same model can drop 15+ points from Verified to Pro — GLM-5.2 goes 78.7% to 62.1%.

Why do vendor benchmark scores differ from independent runs?

Vendors run their own tuned harnesses, choose sampling settings and retry budgets, and publish selectively; referees like Epoch AI and Vals AI standardize the harness and publish regardless of outcome. Neither number is fraudulent by default — they measure different systems. Treat vendor-only scores as provisional until independently reproduced.

How do I benchmark an AI coding agent on my own codebase?

Collect 10–20 real tasks from your history — repo state, the original ask, an acceptance check — and hold them out of agent context. Run each candidate model or harness against them, grading outcomes and trajectories and tracking cost per solved task. Re-run quarterly and on every switch; twenty private tasks beat any leaderboard.

Sources

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv)
• Introducing SWE-bench Verified (openai.com)
• SWE-bench leaderboards (swebench.com)
• Scale AI — SWE-bench Pro (scale.com)
• Vals AI — independent model evaluations (vals.ai)
• Epoch AI Benchmarking Hub (epoch.ai)
• Epoch AI — the state of RL environments (epoch.ai)
• Best open-source coding models, 2026 roundup (morphllm.com)
• METR — measuring long-horizon AI task completion (metr.org)
• Terminal-Bench (tbench.ai)