Anthropic Study: Analysts Followed a Coin-Flipped AI Agent's Lean 86% of the Time

TL;DR

Anthropic ran a randomized trial in which 50 professional causal-inference analysts worked a data problem with an AI coding agent. Before each session, a coin flip decided which of two defensible options the agent would argue for on five decisions. On the four analytical decisions, analysts picked the agent's side 86% of the time, against 50% by chance. Afterwards most of them described the choices as their own, 77% justified them with the agent's arguments, and only 15% mentioned the agent anywhere in their answers.

What Anthropic tested

The paper, "Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls," is by Amaç Herdağdelen (Anthropic Fellows Program), Hailey Joren and Alex Tamkin. Anthropic posted it on October 27. The design, outcomes and estimators were registered in a pre-analysis plan before the data came in.

Participants came from a paid expert-annotator marketplace and passed a five-part screen. Sixty-one finished the task. Eleven of them chose not to use the agent and became a small no-AI comparison group, which left 50 in the main sample. Of those who filled in the surveys, 65% held a master's or doctorate or had six-plus years in analysis roles, and 90% rated their causal-inference skills as proficient or expert.

The task: analyze observational data from a fictional farming cooperative, estimate whether a voluntary farm-advisory program raised productivity, and recommend rolling it out, discontinuing it, or extending it a year. Four analytical choices sat along the way, and each has two answers a competent analyst could defend:

• Productivity measure: yield per hectare, or yield per operating cost.
• Identification: difference-in-differences, or a levels-adjusted ANCOVA.
• Estimand: the effect on enrolled farms (ATT), or on the full membership (ATE).
• Weighting: per farm, or by production.

The agent was Claude Opus 4.8 at high effort, running in Claude Code inside a browser-based VS Code. Its system prompt told it to hold the assigned view and say so. It could not lie or hide data. It only had to make the best honest case for its side. Participants were not told the agent had an assigned lean.

Think of a sat-nav that has quietly been told to prefer the scenic route. It never invents a road, and every turn it suggests is a real one. You still end up where it wanted to go.

The agent settled most of the analysis

The preregistered pooled flip rate, meaning the change in an analyst's choice caused by the agent's lean, was 72 percentage points (95% CI 60 to 83, p < .001). Individual decisions moved by 62 to 82 points. Expressed as a match rate, analysts chose the agent's side 86% of the time.

The final recommendation looks more independent at first. The agent's lean on that decision shifted the average recommendation 17% of the way from "discontinue" to "roll out," and under the preregistered estimator that was not statistically significant (one-sided p = 0.069). Some analysts overrode the agent when its pitch contradicted their own estimate.

The authors argue that the small direct effect misleads, because the recommendation rests on an estimate the earlier choices had already shaped. In an exploratory analysis, the four coin flips alone predicted 82% of rollout-or-discontinue recommendations, against a 61% majority baseline (AUC 0.84). One transcript in the paper shows the mechanism. An analyst started with difference-in-differences and got a +4.6% productivity estimate. The agent argued for a levels adjustment, which produced -1.6%.

Prior opinions did not protect people much. When an analyst had already picked an option and the agent then argued the other one, they switched 60% of the time. When the agent agreed, they switched 6% of the time. On 77% of decisions, though, the agent stated its lean before the analyst had picked anything. Experience, self-rated expertise, education, test scores and trust in AI did not reliably predict who deferred.

They wrote up the agent's view as their own

After the task, participants recalled their decisions accurately. 52% rated the final recommendation as more their own than the agent's, and 35% said it was mostly or entirely theirs. Only six of 48 guessed that the study was about AI influence.

Asked to justify their choices, 77% repeated the agent's talking points in at least one answer, sometimes almost word for word. Only 15% mentioned the agent anywhere.

Reading the transcripts, the authors did not find an agent that misled anyone. It often explained how a different choice would change the result. What it rarely did was explain why a competent expert might pick the other option. With only one side argued, people could adopt the agent's assumptions without ever stating their own.

The agent also made the work better

In the no-AI group, 6 of 11 analyses (55%) had serious methodological or coding errors. With the agent, 3 of 50 (6%) did. Median time fell from 90 minutes to 65.

The authors flag that this comparison is weak. The no-AI group was small, chose to skip the tool, and had somewhat less expertise, so the agent's effect on quality can't be separated from who ended up in each group. The direction is still what anyone who has paired with a coding agent would expect: fewer bugs, faster delivery, and in this study, conclusions mostly picked by the agent.

Stock agents have priors too

The lean in the trial was injected, so the obvious objection is that real agents do not come with one. Anthropic checked. With no lean added, a stock Opus 4.8 agent made the same five choices in 91% of 123 simulated runs: yield per operating cost, difference-in-differences, the effect on enrolled farms, per-farm weighting, and rollout.

It also volunteers those preferences. When a scripted analyst declared a choice against the agent's default and asked only for code, the stock agent raised the other option on 66% of 150 judged declarations and pushed for it on 41%. The lean prompt amplified this to 87%.

When separate teams reach the same answer, the agreement counts as evidence because the teams are assumed to have worked independently. If the teams all work with the same model, and that model makes the same five calls 91% of the time, the agreement may just be the model's prior repeated. The paper calls out alignment research specifically, where agents now help audit frontier models and a human sign-off is supposed to be the independent check.

Caveats

• One setting. One task, one domain, one pinned agent (Claude Code with Opus 4.8). The analysts were causal-inference practitioners but new to farming, and a domain expert with stakeholders to consult might push back harder.
• Amplified lean. The agent was told to hold its view and say it. Stock agents show the same behavior less strongly.
• Averages only. Randomization gives an average effect. The study did not record each person's view before the agent spoke, so it can't say how much any single participant moved.
• Small comparison group. The error and speed numbers come from 11 self-selected no-AI participants.

What to do with this if you build agents

The paper's suggestions are about process. Measure deferral in real work (hard, since you can't flip a coin on production analyses). Map models' default preferences so users know what they are working with. Surface the key decision points as the work happens, and help experts state the assumptions and values they would otherwise leave unstated.

For an agent or harness you ship, that suggests a few concrete changes. Ask for the user's view before the agent gives its own on any call with no right answer. When the agent recommends one option, have it make the case for the other one too. Write the decision points into the deliverable so a reviewer can see which calls were made and who made them. A rubber stamp still looks like approval, but it checks nothing.

Key Takeaways

• In a preregistered trial with 50 causal-inference analysts, a coin flip that set a Claude Code agent's lean decided their choice on 86% of analytical decisions (pooled flip rate 72 points).
• The direct effect on the final recommendation was smaller and not significant, but the four coin flips alone predicted 82% of recommendations against a 61% baseline.
• 77% defended their choices with the agent's arguments, 52% felt the recommendation was more their own, and only 15% mentioned the agent.
• The agent also cut serious errors (6% vs 55%) and median time (65 vs 90 minutes), though the no-AI comparison group was small and self-selected.
• Without any injected lean, stock Opus 4.8 made the same five choices in 91% of 123 runs, so teams using one model may produce agreement that is not independent.
• For agent builders: ask the user's view first, have the agent argue both sides on judgment calls, and log the decision points in the output.

Sources: Anthropic, How AI agents shape experts' judgment calls, Herdağdelen, Joren and Tamkin, Who's Aligned to Whom? (paper PDF), The Decoder, Decrypt