Glonce

GPT-4o mini agent's PMO score falls from 14.44 to 4.85 once task names are hidden

researchThe Glonce editorsarXiv🔥61Glonce rating61Impact45Novelty60Relevance90Surprise55Players10Credibility78

Recent work reports large gains for LLM agents on low-budget molecular optimization. This arXiv paper asks what actually produces them. The authors measure the contributions of three factors: online optimization, retrievable task information, and permissive oracle landscapes.

The central result is on PMO with a budget of 50 oracle calls. A GPT-4o mini agent's sum AUC is 14.44 in the full setup. When task names and curated reference compounds are hidden, it falls to 4.85, roughly a third of the original. A second condition, called identity-blinded, states only the oracle form (for example, similarity to a hidden reference) and scores 5.39.

The sharpest comparison is a deterministic retrieval-only baseline with no LLM at all. It reaches 12.16, which is 84% of the full agent's score. In other words, most of what the agent achieved in the full setup can be matched by looking up information tied to the task. The authors say a full 23-task 2x2 ablation and a three-seed replication on eight tasks support this interpretation.

The paper then turns to a protein-targeted benchmark with an AutoDock Vina composite oracle. There, a single-edit random walk is described as a strong lower bound. Across three seeds it obtains a composite of 0.636 +/- 0.008 and a docking improvement of -1.11 +/- 0.04 kcal/mol.

As a positive control, the authors show that an agent can still beat a classic method when retrieval is removed: a retrieval-free population agent exceeds REINVENT at a matched 50-call AUC budget, with its strongest result at 100 calls.

From this the authors propose two inexpensive sanity checks: retrieval-safe PMO prompts, and a random-walk floor for docking-based optimization.

Key facts

  • On PMO at 50 oracle calls, a GPT-4o mini agent's sum AUC falls from 14.44 to 4.85 after task names and curated reference compounds are hidden.
  • An identity-blinded condition that states only the oracle form scores 5.39, while a deterministic retrieval-only baseline with no LLM reaches 12.16, or 84% of the full agent.
  • A 23-task 2x2 ablation and a three-seed replication on eight tasks support the authors' interpretation.
  • On a protein-targeted benchmark with an AutoDock Vina composite oracle, a single-edit random walk gets composite 0.636 +/- 0.008 and docking improvement -1.11 +/- 0.04 kcal/mol across three seeds.
  • A retrieval-free population agent still exceeds REINVENT at a matched 50-call AUC budget, with its strongest result at 100 calls.

Why it matters

Reported gains for LLM agents on molecular optimization are only meaningful if the agent is actually optimizing. This paper shows that on PMO at 50 calls, a retrieval-only baseline with no LLM reaches 12.16 against the full agent's 14.44. That makes a large share of the headline score attributable to information about the task rather than to the search itself. The positive control matters too: with retrieval removed, a population agent still beats REINVENT at a matched 50-call budget, so the result is not simply that agents do not work.

Who it affects

Researchers building or evaluating LLM agents for molecular optimization, and anyone reading benchmark claims in this area. The comparisons concern PMO and a docking-based benchmark, so the findings speak most directly to results reported on those setups.

How to use it

The authors offer two inexpensive sanity checks. First, use retrieval-safe PMO prompts, meaning prompts where task names and curated reference compounds are hidden. Second, include a random-walk floor when evaluating docking-based optimization, since a single-edit random walk already sets a strong lower bound there. Any new agent result can be compared against these baselines before it is called a gain.

How solid is it

The PMO comparison rests on a full 23-task 2x2 ablation and a three-seed replication on eight tasks, and the docking random-walk numbers are given with spread across three seeds. The PMO AUC figures (14.44, 4.85, 5.39, 12.16) come without statistical uncertainty. The PMO results are reported for a GPT-4o mini agent; no other LLMs are named for that comparison. The abstract names no authors or institutions. This is a preprint.

Risks and caveats

The paper does not say the original LLM-agent gains were wrong or fabricated; it reports that scores fall when task names and reference compounds are hidden. It also does not name which prior works reported the large gains. The score of the retrieval-free population agent and of REINVENT is not given, only that the agent exceeds REINVENT. The 84% figure is relative to the full agent's score, not an absolute metric.

“These results motivate two inexpensive sanity checks: retrieval-safe PMO prompts and a random-walk floor for docking-based optimization.”

— From the paper's abstract