Cognition's FrontierCode 2.0 Grades Coding Agents on Vague Prompts. Opus 5.5 Tops It at 45.5%, and Max Effort Has a Cost

Cognition announced FrontierCode 2.0 on October 27. It is the new version of its private coding benchmark, and it asks a harder question than “does the patch pass the tests?” In Cognition’s words, the question is not only whether you would merge the PR, but whether you should. The prompts are deliberately ambiguous. The agent has to work out what a maintainer would expect without being told.

The top score is 45.5%, from Claude Opus 5.5. The more useful findings for anyone running Claude Code, Codex or Devin-style agents are about failure modes. Agents still miss work that the codebase implies, and turning reasoning effort up to maximum trades one kind of failure for another.

All results below come from Cognition’s FrontierCode 2.0 write-up unless another source is named. They are vendor-run numbers. The task set is private.

What changed from FrontierCode 1.x

FrontierCode 1.0 had open-source maintainers write grading criteria for mergeability. Version 2.0 keeps that method: more than 20 maintainers each spent over 40 hours per task, building tasks from repositories they maintain. Cognition says version 2.0 adds:

• Tasks from 55% more repositories, including security patches and performance optimizations.
• 43% more blocking criteria. A solution that fails any blocker scores zero.
• Bigger tasks. Compared with FrontierCode 1.1, reference solutions touch 67% more files and 89% more lines, in codebases that are 25% larger.
• Stricter auditing. In 80% of tasks the reference solution differs from the upstream PR, up from 60% in 1.0. One automated audit agent found a security vulnerability in an upstream implementation, which Cognition reported to the maintainers.

The biggest change is thoroughness. This category covers correctness conditions that an agent should infer even though the prompt doesn’t state them. Examples include fixing the same bug in a sibling code path, or supporting the other input types that similar code already handles. Thoroughness blockers grew from 168 in 1.1 to 501 in 2.0. Over the same period, the failure rate on those checks rose from 11.7% to 19.2%. Thoroughness is now the most-failed blocker category, ahead of core behavior at 14.9%.

The scores, and how to read them

The headline number is a weighted aggregate of rubric items. Any failed blocking criterion sets that solution to zero. That makes 45.5% a score, not the share of tasks completed.

Model	FrontierCode 1.1	FrontierCode 2.0
Claude Opus 5.5	54.6%	45.5%
GPT-6 Astra	53.3%	42.8%
Grok 4.7	47.6%	28.6%
Gemini 4 Argon	47.5%	26.3%
Muse Wonder 1.0	47.0%	24.3%
SWE-2	50.0%	23.9%
DeepSeek V4.1 Flash	43.9%	19.3%
Kimi K3	44.2%	17.7%

On 1.1, these eight models sat between 44% and 55%. On 2.0 they range from 18% to 46%. That spread is the clearer separation between capability tiers that Cognition claims for the new version. Cognition’s cost chart uses mean list-price spend per rollout. It puts Opus 5.5 on the high end of the Pareto frontier and Luna, Haiku and Sol models on the low-cost end. Cognition singles out GPT-6.1 Sol for strong performance relative to its cost.

Artificial Analysis also runs the benchmark. It added FrontierCode 2.0 to its Intelligence Index v5 at a 15% weight, as announced on X. It describes the set as 132 private tasks and runs each task three times in its own Stirrup harness. Its numbers broadly match Cognition’s top end: Opus 5.5 (max) at 46%, then GPT-6 Astra, GPT-6.1 Terra and Claude Sonnet 5.5 (all max) tied at 42%. They diverge lower down. Artificial Analysis scores DeepSeek V4.1 Flash 1017 (max) at 28%, against Cognition’s 19.3% for DeepSeek V4.1 Flash. The model build and harness both differ, so treat those two figures as separate measurements, not a contradiction. Our coverage of Princeton’s harness-dependence work explains why that matters.

Max effort fixes omissions and adds overreach

FrontierCode is one of the few benchmarks where writing more correct code can still lower your score. Cognition pooled four models (Opus 5.5, Fable 5.1, GPT-6.1 Sol and GPT-6 Astra) and measured blocker failure rates at each reasoning effort level:

Blocker category	Low effort	Max effort
Thoroughness	26.0%	15.3%
Core behavior	20.9%	12.3%
Backwards compatibility	9.7%	5.2%
Scope	3.8%	4.9%

Higher effort clearly helps the agent find implied work. Scope is the exception: it gets worse. Cognition says models write more code and touch more files as effort rises, even when those changes aren’t wanted. It also reports more cheating at high effort, with models “accidentally” pulling the solution commit, and more overthinking into a misread of the request. Cognition loosened the 2.0 scope criteria after finding some reasonable solutions that had failed scope blockers. Even so, the scope failure rate still rises with effort.

Two of Cognition’s task examples show what these failures look like:

• Mock Service Worker, an omission. Opus 5.5 fixed resetHandlers() by dropping the reference to the old generator. It never closed that generator, so the user’s finally cleanup code never runs. The prompt didn’t mention cleanup. It passed 0 of 5 runs at every effort level.
• PostHog Android SDK, overreach. GPT-6 Astra passed 5 of 5 runs at low effort. It passed only 1 of 5 at max effort, where it often deleted PostHogSendCachedEventsIntegration.kt. That class is part of the SDK’s public API, so deleting it is a breaking change. A scope blocker caught it.

What to do with this

• Put implied constraints into your instructions. The thoroughness failures map directly to lines you can add to AGENTS.md or CLAUDE.md: “fix every instance of a bug, not just the reported path,” “match the input types sibling code already supports,” “close or clean up any resource you discard,” and “never delete or rename public API.”
• Don’t set max effort by default. Pick the effort level per task type and measure it per completed task. Our guide to pricing effort per completed task shows the method. Max effort appears to pay off on bug hunts where omissions are the risk. On narrow fixes to a public surface, it is more likely to over-edit.
• Add a scope check to review. A diff-path allowlist or a check for changes to public API is a cheap way to catch the PostHog-style failure before a human reads the PR.
• Use the cost frontier. If a cheaper model on Cognition’s frontier, such as GPT-6.1 Sol, does well enough on your own tasks, save the expensive model for work where thoroughness failures cost the most. See cost per completed task vs token price.

You can’t run FrontierCode yourself, because the tasks are private to avoid contamination. Its categories still work as a review checklist. For broader context on reading agent leaderboards, see When SWE-bench Saturates.

Load X post

Loading this post connects to X.

The original source is available above.

Load X post

Loading this post connects to X.

The original source is available above.