Anthropic warns AI tools could help design bioweapons
MIT Technology Review's biotech newsletter The Checkup examines whether AI could help someone design and release a bioweapon, prompted by a wave of public warnings from AI industry leaders. Anthropic CEO Dario Amodei argued last weekend that AI carries serious risk and that progress should be slowed; OpenAI CEO Sam Altman responded on X, "I agree with Dario that we need to pace the frontier." Days earlier, AI researcher Jacob Coxon announced he was leaving a role at Anthropic (he had also worked at OpenAI), charging that neither company was acting responsibly: "The people building AI earnestly believe that it could kill us all by the end of the decade." Anthropic employee Evan Hubinger publicly agreed, writing on X, "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade."
The article grounds the bioweapon fear in a concrete precedent: in 2022, researchers at Collaborations Pharmaceuticals had built an AI "molecule generator" to find candidate drugs for human disease. Repurposed to search for toxicity instead of safety, it generated 40,000 molecules with potential as chemical warfare agents in under six hours, some more toxic than known nerve agents. The researchers themselves wrote that this "should serve as a wake-up call for our colleagues in the 'AI in drug discovery' community." Stanford professor of medicine and biomedical ethics David Magnus, who has studied biotechnology misuse risks since the late 1990s, calls that finding "very scary" and says "everything since then has just sort of blown up."
Today's large language models compound the concern: Dunja Sabra, a biosecurity researcher at the University of Hamburg, notes anyone can query models trained on the knowledge of "almost every scientist who ever lived on this planet" for experiment instructions and video training, while cheaper gene-editing and synthetic-biology tools have fueled a "DIY biology" movement of home labs. "The chances are that someone determined would succeed eventually," Sabra says. Existing safeguards include screening by companies that sell synthetic DNA, "red-teaming" and "blue-teaming" reviews of risky research, and AI companies restricting what their models will say, but the article states plainly that none of these protections are ironclad. In a report published last week, Anthropic acknowledged that people had tried to use its models to make the chikungunya virus more transmissible, engineer a more dangerous form of bird flu, and build an "atlas of venom toxin peptides," among other attempts. "We've got a constant back and forth," Magnus says. "We have to build better surveillance and screening tools, [but] AI is really good at figuring out ways around them." MIT biologist Kevin Esvelt added on X that a large language model had "disclosed a novel form of bioweapon that I hadn't realized was possible," urging caution.
Not every expert shares the alarm. Some biologists at Imperial College London argued at a recent media briefing that AI tools are not yet capable of fully developing bioweapons, since testing new pathogens still requires difficult, time-consuming human lab work, and that existing guardrails may be sufficient. Imperial's Wendy Barclay went further, arguing the bigger pandemic risk isn't a bioweapon at all but pathogens already circulating, pointing to H5N1 bird flu, which has killed millions of birds, spread through US dairy cattle, and was detected in captive mink at a Utah farm last month. Sabra, looking five to ten years ahead, says countries should be strengthening health-care systems, preparing antidotes to known toxins, and stockpiling medicines now.
Key facts
- In 2022, an AI "molecule generator" built by Collaborations Pharmaceuticals to find drug candidates generated 40,000 molecules with potential as chemical warfare agents in under six hours, some more toxic than known nerve agents.
- Anthropic CEO Dario Amodei said AI progress should be slowed over serious risk; OpenAI CEO Sam Altman agreed on X, and Anthropic's Evan Hubinger put the odds of AI killing all humans within a decade at over 10%.
- Anthropic's own report last week acknowledged attempts to use its models to make chikungunya more transmissible, engineer a more dangerous bird flu, and build an "atlas of venom toxin peptides."
- Safeguards, DNA-order screening, red- and blue-teaming, and model restrictions, are described as not ironclad; researchers David Magnus and Dunja Sabra say AI keeps finding ways around them.
- Imperial College London biologists counter that AI can't yet fully develop bioweapons because pathogen testing remains slow, hands-on work, and Imperial's Wendy Barclay says already-circulating pathogens like H5N1 pose the bigger pandemic risk.
Why it matters
The alarm here isn't hypothetical: Anthropic's own report states people already tried to use its models to make a virus more transmissible, engineer a more dangerous flu strain, and assemble a toxin database. Paired with the 2022 finding that a drug-discovery AI could be flipped into generating 40,000 candidate chemical weapons in under six hours, and Kevin Esvelt's claim that a language model disclosed a bioweapon method he hadn't known was possible, the piece argues that AI-assisted biological risk has moved from theory to observed attempts, even if none is known to have succeeded.
Who it affects
Biosecurity and biotech researchers, the AI companies building and restricting these models, the labs and companies that screen DNA-synthesis orders, and policymakers weighing how tightly to regulate both AI and synthetic biology. Ultimately it's a public-health question: the piece frames it as a risk to anyone exposed to a future engineered pathogen or a more capable natural one.
How to use it
The article names the safeguards currently relied on: DNA-synthesis companies screening orders for suspicious requests, "red-teaming" (independent scientists probing research for misuse potential) and "blue-teaming" (developing mitigations) reviews of risky work, and AI companies tuning their models to withhold misusable scientific detail. None of it is described as sufficient on its own, David Magnus frames the situation as a continuous arms race, and Dunja Sabra argues the practical response is to prepare now: stronger health-care systems, antidotes to known toxins, and medicine stockpiles built five to ten years ahead of need.
How solid is it
The claims come from named, on-record sources: Anthropic's own report, public X posts from Amodei, Altman, Coxon and Hubinger, and interviews with academics Magnus, Sabra, Barclay and Esvelt. The central data point, 40,000 candidate chemical-warfare molecules generated in under six hours, traces to the Collaborations Pharmaceuticals researchers' own published account. That said, several details are unverifiable from the article alone: dates for the warnings are relative ("last weekend," "last week," "Wednesday") rather than exact, Anthropic's report doesn't state how many misuse attempts it detected, and Esvelt doesn't name which model disclosed the bioweapon method he describes.
Risks and caveats
The article never describes an actual mechanism by which an AI tool would let someone manufacture or release a bioweapon, only that AI could aid the design, creation or release, and no real-world case of an AI-enabled bioweapon is cited. Expert opinion is split: Imperial College London biologists argue current AI tools aren't good enough to fully develop a bioweapon because pathogen testing remains slow, hands-on lab work, and some think existing guardrails already suffice. Wendy Barclay goes further, arguing that already-circulating pathogens like H5N1, not hypothetical bioweapons, pose the greater pandemic risk today.
“Please, for the love of God, children, the future of humanity, or whatever you consider holy, let's err on the side of caution here.”
— Kevin Esvelt, MIT biologist, on X