Glonce

Copy-Paste Language Models add a lightweight copy head to LLMs

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact45Novelty60Relevance100Surprise30Players10Credibility75
Copy-Paste Language Models add a lightweight copy head to LLMs

A new paper on Hugging Face starts from a common assumption: Transformers are intuitively considered to be efficient at copying from their inputs. The authors say they show something different, that the ability of LLMs to copy in context can be improved through explicit modeling.

Their proposal is Copy-Paste Language Models (CPLMs). A CPLM adds a lightweight Copy-Paste head to the conventional language modeling (LM) head, the output layer that normally predicts the next token.

The authors claim that CPLMs improve training efficiency and downstream performance, and that they offer new operating points in long-context and retrieval setups. To test the idea they pretrained models from 135M to 1.7B parameters. They report improvements across all scales and backbones, with negligible training and inference overheads.

The paper also offers an explanation for the gains. According to the authors, CPLMs partially mitigate the LM head gradient bottleneck through an alternative gradient path. In other words, the copy head gives the training signal a second route, which the authors credit for the success of the method.

Key facts

  • Copy-Paste Language Models (CPLMs) add a lightweight Copy-Paste head to the conventional language modeling (LM) head.
  • The authors argue that in-context copying in LLMs can be improved through explicit modeling, even though Transformers are intuitively considered efficient at it.
  • Validation used pretrained models from 135M to 1.7B parameters, with improvements reported across all scales and backbones.
  • The authors report negligible training and inference overheads, plus better training efficiency, downstream performance and new operating points in long-context and retrieval setups.
  • Their explanation: CPLMs partially mitigate the LM head gradient bottleneck through an alternative gradient path.

Why it matters

Copying text from the context is a basic skill for language models, and the paper challenges the idea that Transformers already do it well enough. If an explicit copy mechanism helps, it points to a cheap architectural addition rather than a new model design. The authors also tie the gains to the LM head gradient bottleneck, which links a practical feature to a training-dynamics argument.

Who it affects

The work is most relevant to researchers who pretrain language models and to teams building long-context and retrieval-heavy systems, since those are the setups where the authors say CPLMs offer new operating points. The tested models are small to mid-sized, from 135M to 1.7B parameters.

How to use it

This is a research preprint, not a product. The idea is to add a lightweight Copy-Paste head alongside the standard LM head when pretraining. The text mentions no code or model release, so anyone wanting to try it would need to work from the paper itself.

How solid is it

The claims come from the authors' abstract. They say the approach was validated by pretraining models from 135M to 1.7B parameters, with improvements across all scales and backbones. The text gives no benchmark scores, percentage gains or overhead figures, and it does not name the benchmarks, baselines or backbones. The gradient-bottleneck account is the authors' own explanation of why the method works.

Risks and caveats

The largest model tested is 1.7B parameters, and models larger than that are not mentioned, so the results say nothing yet about frontier-scale systems. With no numbers in the text, the size of the gains cannot be judged. The claim of negligible overheads and the gradient-path explanation both rest on the authors' own reporting.

“Transformers are intuitively considered to be efficient at copying from their inputs.”

— Opening line of the paper's abstract