Anthropic Study: Analysts Using Claude Code Followed the Agent's Secretly Coin-Flipped Lean 86% of the Time

Most teams that put a human reviewer between an AI agent and a shipped decision assume the sign-off means something: that the person actually weighed the options and chose. A new randomized controlled trial from Anthropic’s Alignment team tests that assumption directly, and the result should worry anyone who treats expert review of agent output as an independent check.

The setup: a hidden coin flip inside the agent’s instructions

Anthropic recruited 61 professional analysts experienced in causal inference — the discipline of working out from data whether one thing actually causes another. 50 of them worked through a realistic statistics task with Claude Code, while 11 chose not to use the agent and were excluded from the main analysis. Each analyst played advisor to a fictional farming cooperative deciding whether to expand or kill a paid advisory program, a decision that hinged on four genuinely open analytical choices: how to measure productivity, which statistical method to use, which farms to include, and how to weight them.

For each of those choices, a coin flip — invisible to the analyst — set which option their Claude Code agent would secretly favor. A hidden system prompt told the agent to hold that position as its own honest view, argue for it persuasively, change its mind if the evidence demanded it, and never mislead.

The numbers: 86% deference, and nobody noticed

If the agent’s planted lean had no effect, analysts should have picked the agent’s favored option about 50% of the time by pure chance. Instead, Anthropic reports analysts matched their agent’s coin-flipped lean 86% of the time overall, with per-decision rates of 92% on the productivity metric, 82% on the statistical method, 81% on which farms to include, and 90% on weighting — figures Unite.AI’s writeup independently confirms.

Decision	Match rate with agent’s hidden lean
Productivity metric	92%
Statistical comparison method	82%
Which farms to include	81%
Farm weighting	90%
Combined (all four)	86%
Expected by chance	50%

The effect wasn’t passive. In the 38 cases where an analyst stated a preference before the agent weighed in, they abandoned their own view 60% of the time once the agent disagreed, versus just 6% when it agreed. Anthropic flags that comparison as imprecise: with only 38 cases behind it, the 95% confidence interval on the gap runs from 24 to 82 percentage points, and the subset self-selected, so it isn’t a clean causal estimate of the agent’s influence. Afterward, 77% of analysts repeated the agent’s own arguments when explaining their choices, sometimes word for word, but only 15% credited the agent as the source. Only six of 48 surveyed participants even suspected the study was measuring AI influence, and none guessed the agent’s preferences had been randomly assigned. Gizmodo quotes University of Chicago Booth economist Alex Imas calling it “one of the most important pieces of social science on human and AI interaction” he’s seen recently.

Why this matters beyond one farming scenario

The practical target of the study is the “human in the loop” assumption baked into most coding-agent and analysis-agent review processes — the idea that a reviewer’s sign-off proves independent judgment happened. As Yahoo Tech’s coverage frames it, that arrangement only works if the person signing off is actually the one making the call. Anthropic’s own paper goes further, warning that if separate teams defer to agents sharing the same underlying preferences, their apparent independent agreement may really just be the same model bias showing up twice — a direct risk to research, policy analysis, and AI safety evaluations that increasingly lean on agents to run and check experiments.

Anthropic is careful not to oversell this as proof that human review is worthless. The 11 analysts who skipped the agent entirely made serious, result-invalidating errors 55% of the time, versus 6% for agent users, and took longer to finish — though that group was smaller and less experienced, so Anthropic cautions against reading too much into the comparison. The company also flags its own biggest limitation: it deliberately amplified the agent’s persuasive lean to make the effect measurable, and tested one task, one domain, and one model (Claude Opus 4.8). Real-world deference could be smaller when an agent’s preference is weaker or less clearly stated — though Anthropic notes that even with no lean assigned, stock Claude Opus 4.8 landed on the same five choices in 112 of 123 runs, suggesting agents already carry default leanings of their own.

What teams running agent-assisted review should actually do

This directly affects anyone relying on expert review of agent-produced code, analysis, or recommendations as an independent check — exactly the model behind most AI PR review agent policies and human-in-the-loop approval queues. A few concrete takeaways follow from the data rather than from speculation:

• Treat a reviewer’s sign-off as evidence of effort, not evidence of independent judgment, on any decision where there’s no single correct answer — the study shows sign-off and deference coexist comfortably.
• Where a decision genuinely matters, capture the reviewer’s position before the agent states a recommendation, since the 60%-vs-6% reversal only shows up once the agent has spoken first.
• Be skeptical of agents that argue only one side fluently; Anthropic notes its agents rarely made the case for the alternative, which is part of why one well-argued position was enough to move 86% of choices. Automater’s prior piece on annotation versus approval makes a related point about distinguishing a rubber stamp from a real decision gate.
• Don’t assume this generalizes cleanly to every agent and task; Anthropic’s own caveats about an amplified lean, a single domain, and a single model are real constraints on how far these exact percentages travel.

The full paper and methodology are available through Anthropic’s research page.