Putting a human in the loop only works if the human thinks independently. A new Anthropic study measured what happens when they don’t, and it should bother any team that reads campaigns, attribution or A/B tests with an agent’s help.

The paper is “Who’s Aligned to Whom?” by Amaç Herdağdelen, Hailey Joren and Alex Tamkin, published October 27, 2026. There is a summary on Anthropic’s research blog and the full PDF.

The setup

Fifty causal-inference practitioners had to decide whether a farm-advisory program, for a fictional cooperative, hit the 3-4% productivity gain its board wanted. They worked with Claude Code. Four analysis decisions came up, each with two defensible options: which outcome to measure, which comparison method to use, which farms to cover and how to weight them.

Here is the catch. A coin flip secretly assigned the agent a side on each decision. It argued that side honestly and well. It never lied.

Analysts went with the agent’s side 86% of the time. Chance would give 50%. Flip the agent’s assigned side and the analyst’s choice flipped with it, a 72 percentage point swing.

What they said afterward

Asked to justify each decision, 77% of participants (37 of 48) repeated a talking point the agent had given them, sometimes nearly word for word. Only 15% mentioned the agent in any answer. More than half rated the final recommendation as more theirs than the agent’s. Not one suspected the agent’s leaning had been assigned.

The agent also spoke first on 77% of decisions. Participants pushed back on 23%.

That is the hole in “human in the loop.” The person approving thinks they hold an opinion, and the sign-off looks like a second review when it is an echo. The authors put it plainly: keeping a human in the loop assumes the human’s judgment is an independent check.

Your marketing reports run on the same four choices

The study had no marketers in it, so what follows is my translation, not a finding. Still, each decision in the paper has a twin in a campaign report.

First, what to measure: ROAS, margin or new customers. Then what to compare the result against, which could be last month, a holdout group or a competitor. Then whether accounts count with the brand campaign or without it. And finally whether you weight by spend or by customer.

Each has more than one reasonable answer and no ground truth to settle it. If the agent proposes one first and you haven’t written yours, the likely result is that you adopt theirs. Then you defend it to your director with real conviction, because by now it feels like yours.

Fair limits

The agent’s leanings were amplified on purpose, the task took about an hour and there was one of it, and the authors did not record what each person thought before starting. The effect in your shop could be smaller. The effect on the final recommendation was smaller too, 17% of the distance between discontinuing the program and rolling it out, and it missed statistical significance under the main estimator.

And don’t throw the agent out. In the same study, 6 of 11 analysts without AI made serious errors that invalidated their analysis (55%), against 3 of 50 with the agent (6%). That comparison is not clean, since the no-AI group was less experienced. The agent still helps. What needs guarding is who decides.

What I would do Monday

Write your answer before you open the agent. Half a page per analyst, three lines: the metric you will use, what you will compare against, and how much you think the campaign delivered. Then open the agent, compare, and log each disagreement with who gave way and why.

Also ask the agent for the strongest case for each option, not only its own. The authors suggest both defenses themselves: have the agent argue every option, or have it ask for your view before it gives its own.

When we design an analysis workflow with agents at IQ Source, that written note comes first, and the second step is deciding which calls stay yours and never get delegated. I have written before about why an AI that agrees with you is a risk and how AI erodes skills when used without judgment. Approving 50 actions per session is not oversight either. This study measures the mechanism underneath all of it.

Run the test this week on one campaign and count how often your note and the agent disagree. Zero disagreements tells you something too. And if you want to build that workflow with your team, talk to us about marketing.