Glonce

IAR post-training framework improves retrieval-free document QA in LLMs

researchThe Glonce editorsHugging Face🔥78Glonce rating78Impact65Novelty72Relevance98Surprise38Players18Credibility88
IAR post-training framework improves retrieval-free document QA in LLMs

Large language models often cannot answer questions about a bounded set of documents unless those documents are retrieved and placed in the context window at inference time. The authors describe this task as document knowledge internalization: turning a fixed corpus into knowledge the model carries in its own parameters, so it can answer questions about that corpus without retrieval-augmented generation.

To address it, they propose IAR (Inject, Align, Recover), a three-stage post-training framework. Inject converts the source documents into continuation, rewrite, and instruction-conditioned reconstruction training objectives, rather than relying on conventional continued pretraining. Align then adapts the injected model using answer-only question-and-answer supervision, teaching it to respond in the right format. Recover merges the domain-adapted model back with the base instruction model, to restore general-purpose capabilities that can otherwise degrade during the first two stages.

The authors test IAR on the Common Corpus (CC) and CCI document collections, across four model families: Llama, Phi, Qwen, and SmolLM. In the main comparison against Vanilla SFT (standard supervised fine-tuning), IAR improves on all four reported metrics in 7 of 8 dataset-model settings tested. Averaged across these settings, IAR gains 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance, the latter measured across the IFEval, MMLU, and MSBench benchmarks, compared to Vanilla SFT.

In extended tests on CC, the authors also compare IAR against LoRA and FAPM baselines. They report that LoRA and FAPM can each win on individual general-performance metrics, but among the methods that also reach leading or near-leading domain internalization scores, IAR keeps one of the strongest general-capability profiles overall.

Key facts

  • IAR (Inject, Align, Recover) is a three-stage post-training framework that converts a fixed document corpus into parametric knowledge, for retrieval-free question answering.
  • It is tested on the Common Corpus (CC) and CCI datasets, across four model families: Llama, Phi, Qwen, and SmolLM.
  • Against Vanilla SFT, IAR improves on all four reported metrics in 7 of 8 dataset-model settings.
  • Average gains over Vanilla SFT: 3.6 percentage points in domain QA accuracy and 12.1 percentage points in mean general performance across IFEval, MMLU, and MSBench.
  • In extended CC baselines, LoRA and FAPM can win individual general metrics, but IAR keeps one of the strongest general profiles among methods that also reach leading domain internalization.

Why it matters

Retrieval-augmented generation is the default way to give a model access to a fixed set of documents, but it needs a working retrieval pipeline at query time and adds latency and infrastructure. IAR targets the alternative: baking a bounded corpus directly into a model's parameters, so the model can answer questions about it without retrieving anything at inference time. The paper's central claim is that a staged post-training recipe, rather than a single continued-pretraining pass or plain supervised fine-tuning, comes closer to preserving a model's general-purpose ability while it absorbs document-specific knowledge.

Who it affects

Teams building question-answering systems over closed or internal document sets, and researchers working on continued pretraining and knowledge-injection methods for language models, are the direct audience. The comparisons against Llama, Phi, Qwen, and SmolLM suggest the method is meant to generalize across different base model families rather than being tuned to one.

How to use it

The text lays out the three stages at a conceptual level: Inject builds continuation, rewrite, and instruction-conditioned reconstruction objectives from the source documents; Align follows with answer-only QA supervision; Recover merges the resulting model with the base instruction model. The text does not mention a code or model release, so there is nothing published to run from the paper itself.

How solid is it

The method is evaluated on two document collections, Common Corpus (CC) and CCI, across four model families, and compared against Vanilla SFT as the main baseline plus LoRA and FAPM in extended CC tests. The headline result, improvement on all four reported metrics in 7 of 8 dataset-model settings, is a fairly broad sweep, and the reported gains are consistent in direction (positive for both domain accuracy and general performance) rather than a single cherry-picked number.

Risks and caveats

The text reports only percentage-point gains over Vanilla SFT, not the absolute domain-accuracy or general-performance scores, so the underlying baseline strength is not visible here. One of the 8 dataset-model settings did not see improvement on all four metrics, and the text does not say which one or by how much it fell short. No author names, institutional affiliations, publication venue, or date are given in the text, and no code or model release is mentioned, so independent verification beyond the paper page is not possible from what is available.