Coding Agent Index v2.1: Scores Fall About 5 Points Under Merge Grading, and Cost per Task Spans 30x

Artificial Analysis has rebuilt the yardstick many operators use to pick coding-agent configurations. Coding Agent Index v2.1, announced Nov. 14, swaps in a much harder agentic-coding benchmark and a hardened terminal benchmark, re-scores every model-and-harness combination, and adds the number that actually decides fleet budgets: cost per completed task. The headline results move a lot, and the pricing spread is now roughly 30x between configs that score within a few points of each other.

What changed in v2.1

The index measures models inside agent harnesses — Claude Code, Codex, Antigravity CLI, Cognition’s Devin Fusion CLI — rather than raw model API scores. Version 2.1 makes two benchmark swaps:

Benchmark	Replaces	What changed
FrontierCode-AA 2.0	FrontierCode-AA 1.1	Artificial Analysis’s implementation of Cognition’s FrontierCode 2.0: 132 real-world coding tasks graded on whether a repository maintainer would actually merge the agent’s change
Terminal-Bench 4.1	Terminal-Bench 4.0	The same 66 tasks, scored with updated verifiers that improve reliability and resist reward hacking

SWE-Atlas-QnA-AA stays in the index unchanged. The publisher’s index page confirms the v2.1 composition: FrontierCode-AA 2.0, Terminal-Bench 4.1, and SWE-Atlas-QnA-AA.

The merge-grade bar is the real story. A change that passes unit tests but would not survive maintainer review now fails. That is closer to what operators actually pay for — a merged change — and it is why scores fell.

The scores: a tie at 58, and almost everyone drops

Claude Fable 5.2 and Claude Opus 5.5 now share the lead at 58. Fable 5.2 hits 58 in Claude Code at both xhigh and high effort. Opus 5.5 (max) scores 58 in Claude Code, and — paired with GPT-6 Luna High in Cognition’s multi-model Devin Fusion CLI — also reaches 58, echoing Devin Fusion’s own multi-model claims.

Two details matter more than the tie:

• Fable 5.2 (max) scores 56, not 58. More of its attempts hit task time limits at max effort. More effort is not free, and on merge-graded real repositories it can actively cost score — consistent with what Cognition found on FrontierCode 2.0.
• Nearly every configuration dropped about 5 points on average versus v2.0, driven by the harder FrontierCode-AA 2.0. The only exception was Codex with GPT-6.1 Terra (max), which stayed within 1 point of its prior score. If your internal leaderboards were tuned to v2.0-era numbers, they are now stale.

Claude Sonnet 5.5 (max, 56) in Claude Code and Gemini 4 Carbon (high, 55) in Antigravity CLI round out the leaders.

The 30x cost spread

The operator-facing number is cost per completed task. Among configurations scoring 48 or above, the spread is roughly 30x:

• Cheapest: Codex with GPT-6.1 Sol (max) at $1.51 per task — scoring 52.
• Most expensive: Claude Code with Fable 5.2 (max) at $44.22 per task — scoring 56.

Read that carefully: the most expensive lane scores four points higher than the cheapest one. Four points of index score is currently priced at a roughly 29x multiplier. Whether that trade is worth it depends entirely on the value of a merged change in your repository — for routine refactors, the $1.51 lane likely wins; for a hard blocker only the top lane closes, and a failed cheap attempt still costs money.

What to do with this

1. Re-baseline your model choice. Average scores fell about 5 points under merge grading. If your go-to config looked good on v2.0 numbers, check its v2.1 row before renewing that assumption.
2. Price the finished job, not the token. This is the same argument AWS makes in its NeMo Switchyard routing walkthrough: a task’s cost depends on turns, replayed history, and retries, so routing decisions should be judged by cost per solved task. The AWS sample’s benchmark runner prints RUNS, SOLVED, COST/TASK, and COST/SOLVED per model group for exactly this comparison — a pattern worth copying on your own workload even if you never touch Bedrock.
3. Watch effort ceilings. Fable 5.2 losing two points at max because of task time limits is a reminder to test your effort setting, not just your model. Timeouts burn budget without producing merges.
4. Split your lanes. A 30x spread is an argument for routing by task difficulty: cheap lane for the 80 percent of tasks any config solves, frontier lane reserved for the ones that resist.

Caveats

These are Artificial Analysis’s benchmark tasks and harness configurations, not yours. A 52-scoring config at $1.51 is not automatically a bargain if its failure mode on your codebase is silent near-misses that a maintainer has to catch. Treat the index as a shortlist generator, then run your own cost-per-solved-task measurement before committing a fleet budget to one lane. For a model of how to read a fresh Artificial Analysis index critically, see how we covered the Decision Index v2 and its calibration scores.

Load X post

Loading this post connects to X.

The original source is available above.