Glonce

Paper Pilot locks LLM citations to evidence, drives fabrication to zero

researchThe Glonce editorsarXiv🔥74Glonce rating74Impact68Novelty72Relevance98Surprise60Players40Credibility82

A paper introduces Paper Pilot, a human-in-the-loop expert system meant to govern how large language models help write scientific manuscripts. The authors argue that existing autonomous discovery and manuscript-generation systems have a governance gap: ideas, methods, results, and claims can flow through AI-assisted workflows without mandatory human approval or any way to trace a claim back to the evidence behind it. Paper Pilot adapts a methodology called Collaborative Agent Reasoning Engineering (CARE) to close that gap. The framework defines eight approval gates across the idea-to-claim pipeline, sets explicit no-pass criteria, classifies claims, keeps an audit log, adds an advisory LLM review step, and locks revisions to approved evidence. It also separates literature-grounded claims from artifact-grounded claims and requires that every reported number and interpretation stay traceable to evidence a human has approved. Its system prompt is released openly, so it can be deployed inside ChatGPT, Gemini, Claude, or an institution's own LLM setup. As a first empirical test, the authors evaluated only the citation-grounding layer, using a controlled, mechanically scored benchmark with two commercial LLMs writing about real arXiv papers, with no LLM acting as judge. Under coverage pressure, drafters running without Paper Pilot's gates fabricated up to 25% of their citations and never once flagged that they were missing supporting evidence. The same two models running under Paper Pilot's evidence-locked rules produced zero fabricated citations and instead surfaced the gaps that had been deliberately planted in the test, marking them as explicit placeholders rather than inventing a source. The authors report that preliminary results for result grounding, revision handling, and adversarial robustness point the same direction, but say the full evaluation of those parts is left to future work. They frame Paper Pilot's goal as making LLM-assisted writing a controlled, human-supervised decision process rather than a fully autonomous authorship pipeline.

Key facts

  • Paper Pilot defines eight approval gates across the idea-to-claim pipeline, with manuscript-owner approval, claim classification, audit logging, and evidence-locked revision control.
  • The framework separates literature-grounded from artifact-grounded claims and requires every reported number and interpretation to trace back to evidence a human has approved.
  • Its system prompt is released openly for use in ChatGPT, Gemini, Claude, or institutional LLM environments.
  • In a controlled, mechanically scored benchmark with two commercial LLMs on real arXiv papers (no LLM judge), ungated drafters fabricated up to 25% of citations under coverage pressure and never flagged the gap.
  • The same models under Paper Pilot's evidence-locked rules produced zero fabricated citations and surfaced the planted gaps as explicit placeholders.

Why it matters

LLM agents are already doing literature analysis, drafting, and review inside scientific workflows, but the tools that let them work autonomously do not force a human to sign off before a claim, number, or citation moves forward, and they give no way to trace a claim back to its source. Paper Pilot is a proposed governance layer for exactly that gap: it adapts the CARE methodology into eight approval gates that a manuscript owner must clear, plus rules that keep every claim tied to evidence someone actually approved.

Who it affects

Researchers and teams using LLMs to help draft, review, or synthesize literature for scientific manuscripts, and any institution weighing whether to let LLM agents touch manuscript work without a human-in-the-loop check. The framework is aimed at applied sciences specifically, where reported numbers and interpretations need to survive scrutiny.

How to use it

The system prompt behind Paper Pilot is openly released, and the authors say it can be deployed directly inside ChatGPT, Gemini, Claude, or an institution's own LLM environment. No separate software, price, or license is described beyond the released prompt itself.

How solid is it

The empirical evidence so far covers only one piece of the framework: the citation-grounding layer, tested on a controlled, mechanically scored benchmark using two commercial LLMs writing about real arXiv papers, with no LLM used as judge. That test found ungated drafters fabricating up to 25% of citations under coverage pressure while never flagging a gap, versus zero fabricated citations and explicit gap-flagging under Paper Pilot's rules. The authors describe results for result grounding, revision, and adversarial robustness as preliminary only, pointing in the same direction but not yet fully evaluated.

Risks and caveats

The paper does not name the authors or their institutional affiliations, does not identify the two commercial LLMs used in the benchmark, does not state a publication venue beyond the arXiv posting, and gives no date for either the system prompt's release or the benchmark's execution. The full evaluation of result grounding, revision, and adversarial robustness is explicitly left to future work, so the zero-fabrication result should be read as validating the citation layer only, not the whole framework.