Princeton's SAgE Group Maps 33 Trends Explaining Why Coding-Agent Benchmark Scores Depend on the Harness, Not Just the Model

When a vendor announces that its coding agent “scores 74% on SWE-bench” or “beats the competition on an internal eval,” the number looks like a property of the model. A new paper from Princeton’s Science of Agent Evaluation (SAgE) group argues that is usually the wrong way to read it — in agent evaluation, the scaffold, tools, environment and compute budget move the score about as much as the model does (sage.cs.princeton.edu/trends/).

“Emerging Trends in AI Agent Evaluation,” published October 19, 2026 by Felix Chen, Stephan Rabanser, Peter Kirgis and Nitya Nadgir at Princeton with Sayash Kapoor and Arvind Narayanan, alongside co-authors from Stanford, the UK AI Security Institute and the Brookings Institution, publishes the paper alongside an interactive explorer for its 33 trends. Narayanan previewed the project back on July 1, 2026 on X, writing that agent evaluation had “quickly become a distinct discipline” and that the group was then still working on the paper (x.com/random_walker). Kapoor and Narayanan’s companion essay on their Normal Technology newsletter, published the same day as the paper, walks through the reasoning in more depth (normaltech.ai).

A benchmark score is a joint product, not a model attribute

The paper’s central methodological claim is direct: model evaluation grades one response against a reference answer, but agent evaluation covers a whole system — model, scaffold and tools acting over many steps — graded on the trajectory, the final state, or both. Its third section spells out the consequence: “the same model and benchmark can score differently depending on who chooses the scaffold, prompts, tools, environment, compute budget, and statistical analysis.”

That is a direct challenge to how leaderboards for coding agents such as Claude Code, Codex, Devin Fusion and Gemini CLI-style harnesses get compared in practice. A model that trails on one scaffold can lead on another, and a vendor benchmarking its own harness against a competitor’s default setup is not comparing models — it is comparing harnesses.

Single runs are not a score; use pass@k and pass-and-k

Because agent trajectories are stochastic, the paper argues a single run is close to meaningless as a score. It proposes reporting both pass@k, where a task counts if any of k runs succeeds, and pass-and-k, where a task counts only if every run succeeds. The gap between the two numbers is informative on its own: a wide gap means the agent is unreliable even where it can succeed at all.

Metric	Counts success when	What a wide gap vs pass@k reveals
pass@k	At least one of k runs succeeds	—
pass-and-k	Every one of k runs succeeds	Agent is inconsistent, not just occasionally wrong

Tool use opens a new contamination surface

Model evaluation already has to worry about training-data contamination — benchmark tasks a model saw verbatim during pretraining. Agent evaluation adds two more surfaces: scaffolds and prompts that bake in knowledge of the test set, and “search-time contamination,” where an agent with a browse or search tool simply retrieves the benchmark’s own answers at inference time. The paper includes a worked example: an agent asked to reproduce a repository’s results table instead searches for and copies the published numbers. The run is scored a success, but the log shows it should not count.

You cannot cherry-pick which variable moved

When a new agent release changes both the underlying model and the scaffold at once, the paper notes you cannot attribute a score change to either component alone — that requires a full factorial design that varies model and scaffold independently and tests for interaction. Many vendor benchmark posts change multiple variables simultaneously and then present the headline score as evidence about the model alone.

Running that kind of rigorous comparison is also expensive: the authors cite one full run of the MirrorCode benchmark costing roughly $50,000 in API credits for a single model, before counting the engineering effort to build and maintain the harness.

Lab capability does not automatically mean field productivity

The paper’s second section pushes past lab benchmarks entirely, citing METR’s randomized controlled trial, which found that developers using agent tools took longer to complete real software tasks than developers who did not — despite expecting a speedup. The authors frame this as a layer-translation problem: evidence about what an agent can do in isolation does not automatically transfer to evidence about how it changes a team’s actual output, and claims pitched at one layer, capability, are routinely used to support claims at another, impact, without the evidence to back it up.

A checklist for reading the next agent benchmark post

Before trusting a leaderboard number for Claude Code, Codex, Devin Fusion, Gemini CLI or any other coding agent, the paper’s framework suggests asking:

• Was the scaffold, prompt set, tool access and compute budget held constant against the comparison agent, or did the harness change too?
• Is the number a single run, or does it report both pass@k and pass-and-k?
• Did the agent have search or browse access that could retrieve the benchmark’s own answers?
• If both the model and scaffold changed at once, does the claim actually isolate which one caused the difference?
• Is a lab capability score being used to imply a field productivity claim it was not designed to support?

None of this makes benchmark scores useless — it makes them legible. For teams choosing between coding agents, our benchmark-reading guide and primer on agent evals cover complementary ground on saturation and measurement design; this paper adds the missing layer of who ran the eval and how. Microsoft’s open-sourced Scope, which tests whether an agent actually uses a specific SDK or MCP server rather than hallucinating around it, is one concrete instance of the harness-level testing the SAgE paper is arguing for.

The authors close with an argument for more participants rather than fewer: “There is a clear window now to bring in new participants, methods, and ideas for agent evaluations, and to meet the pressing need for rigorous evidence in a fast moving space.”

Load X post

Loading this post connects to X.

The original source is available above.