On Oct 27, 2026, Anthropic's Alignment team published a randomized trial with 50 professional analysts using Claude Code. Each agent was given, by coin flip and in hidden instructions, a lean on four analytical choices. The analysts went with that lean 86% of the time. By chance it would be about 50%. My take: the data does not say agents are bad, it says we treat their first opinion as if it were our own.
What exactly did Anthropic measure?
According to the original Anthropic research (Herdağdelen, Joren and Tamkin), 61 analysts were recruited and 50 used the agent (Claude Opus 4.8 in Claude Code). The task: decide whether the advisory program of a fictional farming cooperative worked and should be expanded. For each analyst and each decision, a coin flip set the side the agent would favor, without the analyst knowing. The agent could change its mind if the data turned against its lean and was not allowed to misrepresent results.
| Analytical decision | Analysts who followed the agent's lean |
|---|---|
| How to measure productivity | 92% |
| Comparison method | 82% |
| Which farms enter the calculation | 81% |
| Farm weighting | 90% |
| Overall (4 decisions) | 86% |
Did they notice they were being steered?
Almost never. Of the 48 who answered the final survey, 6 guessed the study was about AI influence, and none suspected the preferences were randomized. Among the 50, 77% reused the agent's arguments in their own explanations, sometimes word for word, and only 15% credited the agent. In 38 cases the analyst had stated a preference before the agent did: when the agent disagreed, they switched sides 60% of the time. When it agreed, 6%. Anthropic itself warns that this estimate is imprecise (95% confidence interval: a gap of 24 to 82 percentage points) because of the small number of cases.
Does this prove coding agents are dangerous?
No, and the authors say so. Three caveats from the paper that change the reading:
- The lean was strengthened on purpose. With no assigned lean, Opus 4.8 made the same five choices in 112 of 123 runs. Real-world effect is likely smaller when the agent does not come with a strong side.
- One task, one field, one model, about an hour. The analysts knew causal inference but were new to agriculture. People who know the domain well tend to defer less.
- The upside is real. 3 of 50 agent users (6%) made a serious error that invalidated the result, versus 6 of 11 (55%) without the agent. But the no-agent group was smaller and less experienced, so Anthropic says the groups are not directly comparable.
One hypothesis from the authors, and the most useful for people who code: experts often do not arrive with a formed opinion, they discover it while weighing options. If the first option on the table is the agent's, it becomes the anchor.
How do you keep the agent from deciding for you?
Ask for both sides before it recommends. This is what the authors suggest as a better agent design (argue the best case for each option, or ask for the human's view first). You can do it today, in the prompt, with any coding agent. Pick the decisions that matter (architecture, library, database schema, migration strategy) and use something like this:
Before implementing, do not recommend yet.
For the decision "database for the job queue service":
1. Argue the best case for option A (Postgres with SKIP LOCKED).
2. Argue the best case for option B (Redis Streams).
3. List what would change my mind on each.
Only after that will I tell you which way to go.Three habits that complement the prompt:
- Write your own preference before reading the agent's. One line on paper. That is exactly the case (prior preference) that switched sides most when the agent disagreed.
- Mark which decisions are yours. If the result carries your name, the highest-judgment choices should not be the ones the agent settled by default.
- Cross-check with a second model. The authors warn that two models with the same preferences produce agreement that is not independent. If two different models diverge, you found a decision that needs you. If they agree, do not conclude you are right, only that there is no obvious divergence.
When is it fine to let the agent decide?
| Type of decision | Who decides |
|---|---|
| Formatting, variable names, boilerplate | Agent, you review the diff |
| Reversible in minutes (local refactor, test) | Agent proposes, you approve |
| Library, architecture, data schema | You, after asking for both sides |
| Security, money, one-way migration | You, with a second model and human review |
This table is my reading, not a result of the study. The research did not test code, and the authors themselves note the effect may differ in other domains.
If you want the second model in the same session, Verboo Code lets you switch models with /model without leaving the terminal.
