CrossProfile benchmark shows LLM group-memory shortcuts fail

When several LLM agents discuss a problem and the discussion is squeezed into a short memory for a final decision maker, what should that memory keep? A new paper, listed on Hugging Face Papers, looks at two natural answers: keep the rarest claims, or keep the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, the authors say, because how widely a fact is held is confounded with whether it helps.

To pull those two things apart, the paper introduces CrossProfile, a diagnostic benchmark of 60 templates (480 tasks in total) with automatically verified answer keys. In it, the ownership of supporting, decoy and irrelevant facts is crossed, while the text of the facts is held fixed. The authors ran it across Qwen2.5-7B, Llama-3.1-8B and two DeepSeek models, and report that both heuristics fail in predictable places.

Relevance selection readmits decoys, because decoys are on topic. The paper's headline number for this: even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration. Rarest-first selection, however, is no better than random selection on average. It wins only where the rare facts happen to be the useful ones. In both cases, the authors write, accuracy follows the balance of evidence the memory keeps, not the policy that kept it.

The decision model adds a third failure, which the authors call a form of social proof: it gives extra weight to facts attributed to many agents. Removing the attribution tags, or reassigning them at random with line order fixed, raises accuracy in the classic configuration by 13 to 24 points.

Finally, in natural discussions the evidence is lost before any memory acts. The complete transcript yields lower accuracy than no notes at all.

Key facts

CrossProfile is a diagnostic benchmark of 60 templates (480 tasks) with automatically verified answer keys; supporting, decoy and irrelevant facts have their ownership crossed while their text stays fixed.
Tested on Qwen2.5-7B, Llama-3.1-8B and two DeepSeek models: both rarest-first and relevance-based memory selection fail in predictable places.
Even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration, because it readmits on-topic decoys; rarest-first is no better than random on average.
The decision model over-weights facts attributed to many agents; removing or randomly reassigning attribution tags raises classic-configuration accuracy by 13 to 24 points.
In natural discussions the complete transcript yields lower accuracy than no notes at all.

Why it matters

Multi-agent LLM setups often end with one model deciding from a compressed record of what the others said. This paper argues that the usual ways of compressing that record rest on a confound: how widely a fact is held in the group is tangled up with whether the fact actually helps. CrossProfile separates the two, and the result is that neither popular shortcut can be trusted. The authors' summary is that accuracy tracks the balance of evidence the memory keeps, not the policy that selected it. They also report that natural discussions lose the evidence before memory even runs, since the full transcript scores below having no notes.

Who it affects

People who build or study multi-agent LLM pipelines where agents debate and a final model decides, especially those who add a summarising or memory step in between. The tested models were Qwen2.5-7B, Llama-3.1-8B and two DeepSeek models, so the findings speak most directly to models of that kind.

How to use it

This is a research paper and a benchmark, not a tool. The one concrete lever it reports: removing attribution tags, or reassigning them at random with line order fixed, raised accuracy in the classic configuration by 13 to 24 points. The paper also gives a diagnostic design to copy in spirit, crossing who holds a fact with whether the fact supports, misleads or is irrelevant while keeping the text fixed. No recommended memory policy is stated.

How solid is it

It is an abstract-level account: 60 templates, 480 tasks, answer keys verified automatically, four named model families or sizes. The source gives only point differences, not absolute accuracy levels for any policy, and it does not say what the 23 to 31 point range spans, per model or across models, or which model produced which value. Only the listed models were reported on.

Risks and caveats

The benchmark is diagnostic and built from templates, so it isolates the effects by design rather than reflecting messy real discussions, though the paper does add a note on natural discussions. The point gaps are quoted without the metric being named in that sentence. The social-proof finding is shown by changing attribution tags in the classic configuration; the source does not say the effects hold for models beyond those tested, and it does not say whether code or data are released.

“accuracy follows the balance of evidence the memory keeps, not the policy that kept it”

— CrossProfile paper abstract, Hugging Face Papers