Discovery Certification Protocol audits AI research agents' claimed discoveries

AI research agents combine prior knowledge, public sources, and experimental feedback to produce results that get reported as useful. But a benchmark score alone does not show that a result is a genuine discovery rather than something an ordinary method could have reached with the same public information. A paper introduces the Discovery Certification Protocol (DCP), which turns a discovery claim into a set of executable recovery and feedback tests instead of resting the claim on its reported score alone.

DCP runs through gates. Gate 1 validates that the result is a useful improvement, tested on a sealed evaluation. Gate 2 gives matched agents the registered starting information and the web content that was actually observed, while withholding the target research history, the specific trail that led to the original result. If any valid method among these matched agents reaches the same numerical target anyway, that supplies what the protocol calls a recovery witness and triggers a Core veto, which disqualifies the claim as recoverable by ordinary means rather than a genuine discovery. Passing DCP Core itself requires adequate controls, zero observed recoveries, and a finite-sample statistical bound on recovery computed from one freshly registered episode.

An optional Gate 3 goes further. It measures the average effect of truthful feedback relative to a specified neutral policy, both run from a shared checkpoint. A second certification tier, DCP Evidence, adds this feedback effect on top of DCP Core, but only after the effect clears an independent null calibration and a registered effect margin set in advance.

Two controlled audits run the complete protocol end to end: one in SQLite optimization, one in virtual catalyst control, each under a different model. Each audit produced zero observed recoveries across 96 episodes, yielding the same finite-sample upper bound of 0.0468 on the true recovery rate. For the optional Gate 3 evidence, each paired study recorded 30 recoveries under truthful feedback against zero recoveries under the neutral policy, calibrated against 60-pair null studies, which passed. Additional test cases beyond the two main audits exercise each of the protocol's three possible outcomes: a Core-certified decision, a recovered, disqualifying decision, and an audit-incomplete decision where the evidence does not suffice either way. A deterministic verifier that uses no LLM reproduces every one of these decisions from the frozen evidence alone.

The paper frames DCP as a common evidence language: a shared way to state whether an AI research agent's outcome is useful, whether an ordinary alternative route could reach it anyway, and what measurable effect truthful feedback had, applicable across different AI research projects rather than argued case by case.

Key facts

The Discovery Certification Protocol (DCP) converts a claimed AI research agent discovery into executable recovery and feedback tests rather than accepting a reported benchmark score as proof.
Gate 2 gives matched agents the registered starting information and observed web content but withholds the target research history; any valid method that still reaches the numerical target supplies a recovery witness and triggers a Core veto that disqualifies the claim.
Two controlled audits, in SQLite optimization and virtual catalyst control under different models, each found zero recoveries across 96 episodes, for a finite-sample upper bound of 0.0468 on the recovery rate.
The optional Gate 3 evidence audit recorded 30 recoveries under truthful feedback against zero under a neutral policy per paired study, calibrated against 60-pair null studies, which passed.
A deterministic, LLM-free verifier reproduces every DCP certification decision, Core, recovered, or audit-incomplete, directly from frozen evidence.

Why it matters

The paper's own title makes the point: scores alone do not prove discovery. AI research agents are increasingly credited with finding useful results on their own, but a headline benchmark number does not distinguish a genuine discovery from a result that any ordinary method could have reached with the same publicly available information. DCP addresses that gap directly: it turns a discovery claim into a concrete, executable test rather than leaving it to rest on a single reported score. By requiring a recovery attempt under matched conditions plus a statistical bound on how often that attempt could succeed, it gives anyone evaluating an AI-driven research result a falsifiable standard to check the claim against.

Who it affects

The protocol targets teams building or evaluating AI research agents that claim to make useful discoveries, in domains like the SQLite optimization and virtual catalyst control cases the paper audits. It also affects anyone who has to review or trust such a claim afterward. A shared, executable certification procedure replaces one-off, per-project judgment calls with a common evidence standard, backed by a verifier a third party can run independently on the same frozen evidence.

How to use it

Applying DCP starts before the agent runs, with the starting information and target registered in advance. Gate 1 checks the result against a sealed evaluation to confirm it is a useful improvement. Gate 2 then hands matched agents that same registered starting information plus the web content actually observed, but not the winning research history, and lets them attempt the same numerical target; any valid method that reaches it produces a recovery witness and triggers the Core veto, disqualifying the claim. Clearing Gate 2 with adequate controls, zero recoveries, and a finite-sample bound from one fresh registered episode earns DCP Core certification. Teams that also want to show their feedback loop itself helps can run the optional Gate 3, comparing truthful feedback against a neutral policy from a shared checkpoint, and clear an independent null calibration plus a pre-registered effect margin to add DCP Evidence certification on top. Because the final verifier is deterministic and uses no LLM, the frozen evidence file can go to a third party for independent re-verification without rerunning the underlying experiments.

How solid is it

The paper backs the protocol with two controlled audits run end to end: SQLite optimization and virtual catalyst control, each under a different model. Both produced zero observed recoveries across 96 episodes. Each yielded the same finite-sample upper bound of 0.0468 on the true recovery rate. The optional Gate 3 evidence held up too: each paired study recorded 30 recoveries under truthful feedback against zero under the neutral policy, calibrated against 60-pair null studies, which passed as expected. Beyond these two main audits, the paper adds test cases built specifically to trigger each of the protocol's three possible decisions, Core-certified, recovered, and audit-incomplete, and confirms that a deterministic, LLM-free verifier reproduces all of them from the frozen evidence alone, a direct reproducibility check on the protocol's own machinery rather than just on the two headline audits.

Risks and caveats

The source text does not name the authors or institutions behind DCP: the HuggingFace submitter listed in the crawl metadata is not credited as an author within the paper's own text, and no publication or submission date is given. The specific models used in the two audits are not identified, and the numerical target or underlying result being audited in the SQLite optimization and virtual catalyst control cases is not described beyond naming the two domains. No timescale is given for how long a fresh registered episode, or the audits themselves, take to run, and the source does not compare DCP against any prior certification approach or say whether it has been adopted anywhere beyond these two demonstration audits. The zero-recovery result is itself a statistical bound, 0.0468, computed from a finite 96 episodes, not a claim that recovery is impossible.