Glonce

IdeaAMBIG benchmark shows LLMs miss gaps in research specs

researchThe Glonce editorsHugging Face🔥68Glonce rating68Impact60Novelty70Relevance88Surprise45Players55Credibility82
IdeaAMBIG benchmark shows LLMs miss gaps in research specs

A team behind a new benchmark called IdeaAMBIG argues that a research idea can be novel, coherent and scientifically sound while its method section is still too vague to build from without guessing. They call this property codification readiness: whether a method specification gives a competent implementer or coding agent enough information to construct it without making unsupported assumptions. To measure it, the researchers assembled evidence-grounded specifications and their supported resolutions by mining papers, codebases, issue threads and reproduction artifacts. The resulting benchmark, IdeaAMBIG, contains 660 evidence-grounded instances: 163 real-world gaps pulled from reproducibility reports and GitHub issues, plus 497 controlled synthetic gaps injected into specifications that were already codification-ready. The benchmark evaluates three separate capabilities. The first is codification-readiness assessment: judging whether a spec is complete enough to implement. The second is defect localization: given only the specification, finding where the missing information is. The third is clarification action generation: given the specification plus the annotated defect, producing the clarification needed to fix it. Across 13 LLMs, the best-performing model recovered only 9.6% of real-world defects on its own using its Macro Defect Recovery Rate, but reached an 80.6% Macro Clarification Action Success Rate once the defect had already been pointed out to it. A separate oracle study underlines the same gap: supplying the gold-standard resolution for a defect raised the downstream codification-ready rate from 14% to 98%. Across every model tested, the bottleneck sits in the same place: finding the gap in the first place, not fixing it once found.

Key facts

  • IdeaAMBIG contains 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references.
  • It tests three capabilities: codification-readiness assessment, defect localization (spec only), and clarification action generation (spec plus the annotated defect).
  • Across 13 LLMs, the best model achieves a 9.6% Macro Defect Recovery Rate on real-world instances.
  • The same best model reaches an 80.6% Macro Clarification Action Success Rate when the defect is already given to it.
  • An oracle study found that supplying the gold resolution lifts the downstream codification-ready rate from 14% to 98%.

Why it matters

A method can pass peer review as novel and scientifically plausible while still leaving a coding agent or a human implementer to guess at critical details. IdeaAMBIG gives that failure mode a name, codification readiness, and a way to measure it instead of treating reproducibility gaps as anecdotal complaints from people who tried to rebuild a paper's method and got stuck.

Who it affects

The benchmark targets LLMs and coding agents being evaluated or built for autonomous or assisted research implementation, plus researchers who write method specifications and reviewers or reproducibility efforts trying to judge whether a paper's description is actually buildable.

How to use it

IdeaAMBIG is structured as three separate tasks that can be scored independently: judging whether a spec is complete, finding where it falls short when it is not, and writing the clarification once a defect has been flagged. That separation lets a team test where its model or agent actually fails in the pipeline, since the text shows the three tasks do not fail at the same rate.

How solid is it

The evaluation spans 13 LLMs and mixes 163 real-world gaps drawn from actual reproducibility reports and GitHub issues with 497 controlled synthetic gaps, rather than relying on synthetic cases alone. The oracle study, where supplying the correct resolution lifts the codification-ready rate from 14% to 98%, is used as a ceiling that shows how much of the problem is solvable once the missing piece is known.

Risks and caveats

The results point to a specific weak spot rather than a general one: the same models that recover only 9.6% of real-world defects on their own turn around and resolve 80.6% of them once the defect is already identified. The hard part is noticing that something is missing, not writing the fix. The abstract does not name any of the 13 models tested or say which one is the best performer, and it gives no detail on how the synthetic gaps were mechanically injected beyond calling the process controlled.

“A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.”

— IdeaAMBIG paper abstract