Glonce

Anthropic study: experts adopt AI agent's random preferences, think they're their own

researchThe Glonce editorsThe Decoder🔥71Glonce rating71Impact60Novelty65Relevance95Surprise60Players70Credibility80
Anthropic study: experts adopt AI agent's random preferences, think they're their own

Anthropic ran an experiment with 50 experts in causal inference to see who really makes the decisions when an AI agent helps with a data analysis. The task was realistic: participants examined observational data from a fictional farming cooperative and estimated the productivity gains from a voluntary farm advisory program. They then had to recommend whether the cooperative should expand the program to all farms or end it; participants or the agent could also suggest extending it for one year.

The catch was hidden from participants. At each of four analytical decision points, such as how to measure productivity or which statistical method to use, a coin toss set the agent's preference for one of two valid options. The agent could not lie or hide data; it only had to make the strongest honest case for its assigned side.

Participants adopted the agent's preference in 86 percent of the four analytical decisions, far above the 50 percent expected if people chose at random. The authors call the agent's net influence the "flip rate", which ranged from 62 to 82 percent depending on the task. Already having an opinion offered little protection: among participants who had made a choice before the agent stated its preference, an opposing view made them 53 percentage points more likely to switch sides than an agreeing one. The authors tested years of analysis experience, self-rated expertise, education, test scores and trust in AI as possible safeguards. None had a statistically reliable link to the results after correcting for multiple tests. Age stood out before that correction, though a sample of 50 gave the study little power to detect such effects.

The agent's assigned preference had only a small direct effect on the final policy recommendation to the cooperative's board, slightly shifting votes without reaching statistical significance, and some experts overruled the agent when its advice did not match their own calculated estimate of productivity. The researchers say this does not show participants kept control: the four earlier decisions had a major effect on the estimated productivity gain, which in turn shaped the board recommendation.

Afterward, 52 percent of participants said the recommendation felt more like their own than the agent's, and another 35 percent said it was mostly or entirely their own. Participants correctly recalled their own choice for 85 percent of decisions, yet 77 percent (37 of 48) used at least one of the agent's arguments in their explanations, sometimes in nearly the same words. Only 15 percent mentioned AI at all. Conversation logs show the agent stated a position first in 77 percent of decisions and then usually argued only for its preferred side, without explaining why skilled experts might choose the alternative, though it did explain how a different choice would affect the result, according to the paper.

AI also helped. Without an agent, 55 percent of analyses (6 of 11) had serious methodological or coding flaws, and the median time was 90 minutes. With AI, 6 percent (3 of 50 analyses) had serious flaws, and participants finished in 65 minutes. The small no-AI group had chosen not to use the tool and had somewhat less expertise, so the researchers cannot say for sure how much of the gap came directly from AI.

The researchers argue that this quiet deference could have wide effects in science. If many research teams use the same model with fixed default preferences, they could end up creating a false consensus. The risk extends to AI safety: researchers already use AI agents to test frontier models for safety and alignment, expecting human review to act as an independent check. If experts unknowingly adopt an agent's bias on open-ended judgment calls, a misaligned system could end up steering the very research meant to expose its misalignment.

The team calls for ways to measure deference to AI agents in real work. Agents should make the best case for both sides, or ask experts for their own view before stating a position. Institutions need to decide which calls require independent human judgment and set aside time and resources for them.

The Decoder places the study among earlier work. A study on false confidence with 3,132 participants found that seeing AI answers nearly erased willingness to withhold judgment: in one part, the share willing to say "I don't know" fell from 44 to 3 percent. With polished AI outputs such as code or documents, users checked facts less often (by 3.7 percentage points), questioned reasoning less often (by 3.1 points) and missed missing context more often (by 5.2 points); users who revised outputs step by step questioned the AI's reasoning 5.6 times more often than those who did not iterate. In a Polish observational colonoscopy study in The Lancet Gastroenterology & Hepatology, experienced doctors' rate of finding precancerous lesions in exams without AI fell from 28.4 to 22.4 percent after regular AI use. A Swiss Business School survey found a strong negative link between AI tool use and critical thinking, mainly among people ages 17 to 25. The Decoder adds that these are early studies and snapshots, and that other results point the other way: a pilot in Nigeria found students using a chatbot as a virtual tutor twice a week made learning gains in six weeks equal to nearly two extra years of learning.

Key facts

  • Anthropic had 50 causal-inference experts analyse data with an AI agent whose preference on each of four decisions was set by coin toss; participants did not know this.
  • Participants adopted the agent's preference in 86 percent of the four decisions (chance would be 50 percent); the agent's net influence, the "flip rate", ranged from 62 to 82 percent by task.
  • 52 percent said the recommendation felt more like their own than the agent's and another 35 percent said it was mostly or entirely theirs, yet 77 percent (37 of 48) reused the agent's arguments and only 15 percent mentioned AI.
  • With AI, 6 percent of analyses (3 of 50) had serious flaws and participants finished in 65 minutes; without it, 55 percent (6 of 11) had serious flaws, with a median of 90 minutes.
  • The authors warn that many teams using the same model could create a false consensus and weaken human review in AI safety research; they suggest agents argue both sides or ask for the expert's view first.

Why it matters

AI agents are increasingly handed analysis work, and the usual comfort is that the AI suggests and the human decides. This experiment says the picture is murkier. The agent's preference was arbitrary, set by a coin toss, yet experts followed it far more often than the 50 percent chance level, and most still felt the result was theirs. Participants recalled their own choices for 85 percent of decisions but often echoed the agent's reasoning, sometimes in nearly the same words. The authors also tie this to AI safety: researchers use AI agents to test frontier models and count on human review as an independent check, but that check weakens if experts unknowingly absorb an agent's bias.

Who it affects

Anyone who uses an AI agent for open-ended analysis where two or more defensible options exist: researchers, data analysts and statisticians, and the teams and institutions that rely on their work. The researchers single out science as a whole, where many teams using the same model with fixed default preferences could produce a false consensus. They also point to AI safety and alignment researchers, who use agents to test frontier models. Experience, self-rated expertise, education, test scores and trust in AI showed no statistically reliable protective link, so seniority does not appear to be a shield, though the sample of 50 gave little power to detect such effects.

How to use it

The study is a finding, not a tool, but the authors give concrete suggestions. Agents should make the best case for both sides, or ask experts for their own view before stating a position; the agent in the experiment stated a position first in 77 percent of decisions. The team also calls for ways to measure deference to AI agents in real work, and says institutions need to decide which calls require independent human judgment and set aside time and resources for them. In the researchers' words: "There may be no single right division of labor between people and AI. Some decisions may be fine to hand over, and others worth keeping." The speed and error gains were real: 65 minutes against a 90-minute median, and serious flaws in 6 percent of analyses against 55 percent.

How solid is it

The core design is clean: a coin toss assigned the agent's side, the agent could not lie or hide data, and the effect sits well above chance. But this is The Decoder's report of the paper. The source does not name the AI model or agent used, and no p-values or confidence intervals are given. The sample is 50 experts, which the authors say gave little power to detect effects of age or similar traits. The no-AI comparison covers only 11 analyses from a group that had chosen not to use the tool and had somewhat less expertise, so the researchers cannot say how much of the gap in errors and time came directly from AI. The related studies cited (false confidence, colonoscopy, Swiss Business School, Nigeria) are separate work, and The Decoder calls these early studies and snapshots.

Risks and caveats

The final policy recommendation was only slightly shifted by the agent, without statistical significance, but the researchers stress this is not proof of human control, since the four earlier decisions shaped the productivity estimate behind it. The study had no clearly right answer by design, and the source does not say the agent's advice was wrong or biased in a harmful direction. The source gives no timescale for the false-consensus risk and no evidence it has happened in real research; it is a stated possibility ("could end up"). The agent could also be honest and still persuasive: it usually argued only for its preferred side. The 86 percent adoption figure and the 62 to 82 percent flip rate are reported separately, and the source does not say whether they are the same measure.

“There may be no single right division of labor between people and AI. Some decisions may be fine to hand over, and others worth keeping. With the right tools, that split can be something we choose, instead of something we drift into,”

— The study authors, as quoted by The Decoder