OpenAI introduces MedChemBench, a 99-problem chemistry test for AI models

OpenAI introduces MedChemBench, a 99-problem chemistry test for AI models

OpenAI has introduced MedChemBench, a benchmark that tests AI models on reasoning tasks drawn from medicinal chemistry research. The aim is to see whether a model can connect a molecule's structure with experimental evidence to explain results, compare compounds and predict their properties. OpenAI frames this as one step towards helping scientists develop better medicines.

The post starts from the central difficulty of the field: improving one property of a molecule can make another worse. A compound might act strongly against a disease-related protein but be poorly absorbed or break down too quickly in the body. Medicinal chemists work through these tradeoffs by designing, making and testing related compounds, then deciding what to try next.

The benchmark has 99 problems drawn from published research, built on 117 papers and patents and 761 distinct molecular structures. For most problems the model gets the context available at a chosen point in a research program: the biological target, the researchers' goals, the compounds studied so far and selected experimental findings. Scoring covers four components. Molecular structures: correct atom connections and specified stereochemistry in molecules assembled from the drawings and chemical groups provided. Scientific reasoning: explanations supported by the supplied evidence, such as why one compound was chosen over another. Ranking and selection: rankings and choices that meet the stated requirements. Property predictions: agreement with the numerical values and categories reported in the source material, with partial credit based on closeness to the measured values. Measurements already supplied in the prompt must also be reproduced correctly.

One example problem comes from research on inhibitors of PRMT5, a protein studied in cancer drug discovery. The model sees molecular structures and earlier results, then must predict how three related compounds behave: how strongly they block PRMT5 in cells, how readily they cross a cell barrier, and how quickly liver enzymes break them down in a laboratory assay. Experiments withheld from the model showed a tradeoff. Compound 23 blocked PRMT5 in cells at a lower concentration, while compound 24 crossed the cell barrier more readily and broke down more slowly in the liver-enzyme assay.

A second example tests structure reconstruction. Models must assemble a molecule from a central structure and two attached groups while preserving an 18-atom ring. At xhigh, the highest reasoning setting shared by all eleven models, GPT-6 Astra got the compound right in all ten responses. GPT-5.6 Sol and GPT-5.6 Luna failed in all ten. In the illustrated answers, GPT-5.6 Sol produced a 15-atom ring and GPT-5.6 Luna split the molecule into two disconnected fragments. OpenAI notes that such errors change the identity of the molecule being analysed.

OpenAI evaluated eleven GPT models at different reasoning-effort settings. In the introduction, at xhigh, GPT-6.2 Astra earned 84.5% of the available credit, against 83.3% for GPT-6 Astra, 82.0% for GPT-6.1 Sol and 58.2% for GPT-5.6 Sol. A later passage on the full benchmark at the shared xhigh setting gives Astra 83.3%, GPT-6.1 Sol 82.0%, GPT-6.1 Terra 75.1% and GPT-5.6 Sol 58.2%. At max effort, GPT-6 Astra reached 83.8%. Higher reasoning settings improved scores, while large differences between models remained.

By component, Astra earned 95.4% of the available credit on structures at xhigh, and its clearest strengths were reconstructing structures and explaining the scientific evidence. In reasoning examples from OpenAI's preprint, Astra correctly identified structural changes and weighed their possible effects on protein binding, absorption and metabolism. Selecting compounds and predicting numerical properties was much harder: only about half the credit. OpenAI adds that these categories use different tasks and scoring rules, but the results show substantial room to improve predictions and compound choices.

OpenAI is explicit about limits. The results do not establish whether models can discover new drugs or improve laboratory decisions. Prior exposure to the published sources cannot be ruled out, and each response was a single attempt without tools, feedback or new experiments. The benchmark evaluates reasoning from historical evidence rather than real-world drug discovery. OpenAI says it believes MedChemBench will support continued progress and plans to share updates as its models advance.

Key facts

  • MedChemBench has 99 problems drawn from published research, using 117 papers and patents and 761 distinct molecular structures.
  • It scores four components: molecular structures, scientific reasoning, ranking and selection, and property predictions, with partial credit for numerical predictions close to measured values.
  • OpenAI evaluated eleven of its GPT models; at xhigh, GPT-6.2 Astra earned 84.5% of the available credit, against 83.3% for GPT-6 Astra, 82.0% for GPT-6.1 Sol and 58.2% for GPT-5.6 Sol.
  • Astra earned 95.4% of the structure credit at xhigh but only about half the credit for selecting compounds and predicting numerical properties.
  • OpenAI says the results do not establish whether models can discover new drugs or improve laboratory decisions.

Why it matters

Medicinal chemistry is a tradeoff problem: a compound that hits its target hard may still be absorbed badly or break down fast. MedChemBench tests whether a model can reason across those tradeoffs from real published evidence, instead of answering isolated chemistry questions. The reported pattern is the useful part. Models did well at reading and rebuilding molecular structures and at explaining evidence, but much worse at the decisions chemists actually face, choosing which compounds to pursue and predicting measured values. OpenAI itself says there is substantial room to improve there.

Who it affects

The post speaks to scientists and chemists who might one day use AI in drug discovery, and to teams building and comparing reasoning models for scientific work. It is also OpenAI's own scorecard: only its GPT models are evaluated, spanning eleven models at different reasoning-effort settings. No models from other labs are reported.

How to use it

The post does not say the benchmark is publicly released or available to outside researchers, nor how it can be accessed. It mentions a preprint with selected reasoning examples but gives no link or title for it. For now it is something to read for its design and results: the four scoring components, the PRMT5 prediction task and the 18-atom-ring reconstruction task show what kind of reasoning is being tested.

How solid is it

This is a first-party announcement: OpenAI designed the benchmark and evaluated its own models on it. The numbers come from the post itself, and no human-expert (medicinal chemist) baseline is reported. The figures also need careful reading. The introduction gives GPT-6.2 Astra 84.5% at xhigh, while the later full-benchmark paragraph speaks only of Astra at 83.3% at the shared xhigh setting, and the text does not reconcile the two. Exact figures for Astra's selection and prediction components are not given, only 'about half the credit', and the data behind Figures 1 to 3 is not in the text.

Risks and caveats

OpenAI states the main caveats itself. The results do not establish whether models can discover new drugs or improve laboratory decisions. Prior exposure to the published sources cannot be ruled out. Each response was a single attempt without tools, feedback or new experiments. The benchmark evaluates reasoning from historical evidence rather than real-world drug discovery. The structure, selection and prediction categories also use different tasks and scoring rules, so the gap between them is not a like-for-like comparison. The post does not claim that AI can identify drug candidates in practice, so its headline question is left open.

“These results do not establish whether models can discover new drugs or improve laboratory decisions.”

— OpenAI, MedChemBench announcement