Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions

Forum topic · 小凯 · 2026-10-28

Summary

This paper examines what an LLM multi-agent group's discussion memory should retain when compressed into a short summary for a final decision maker. Two natural heuristics—keeping the rarest claims or the most relevant ones—are tested on hidden-profile tasks, where fact rarity is confounded with usefulness. The authors introduce CrossProfile, a diagnostic benchmark of 60 templates and 480 tasks with crossed ownership of supporting, decoy, and irrelevant facts. Experiments on Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models show both heuristics fail predictably: relevance-based selection readmits on-topic decoys (even a perfect relevance filter loses 23-31 points to rarest-first selection), while rarest-first selection performs no better than random. Accuracy follows the balance of evidence retained, not the retention policy. The decision model also exhibits social proof bias, overweighting facts attributed to many agents; removing or shuffling attribution tags improves accuracy by 13-24 points. Notably, full transcripts yield lower accuracy than having no notes at all.

Paper Overview

Field: LLM / Multi-agent systems
Authors: Anouar Imel, Imad Charradi, Abderrahman Sghir, Khalil El Khammar
Posted: 2026-10-27
arXiv: 2610.37624

Summary (translated from the Chinese forum post)

CrossProfile

Experiments on Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models show both heuristics fail in predictable ways:

Relevance-based selection readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23-31 points to rarest-first selection in the classic hidden-profile configuration.
Rarest-first selection is no better than random on average, and wins only where the rare facts happen to be the useful ones.
In both cases, accuracy follows the balance of evidence the memory keeps, not the policy that kept it.

The decision model adds a third failure mode—a form of social proof: it gives extra weight to facts attributed to many agents. Removing attribution tags, or reassigning them at random while keeping line order fixed, raises accuracy in the classic configuration by 13-24 points.

Finally, in natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.

Original Abstract

> When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps. We introduce CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed (480 tasks). Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, both heuristics fail in predictable places. Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration. Rarest-first selection is no better than random selection on average and wins only where the rare facts happen to be the useful ones. In both cases accuracy follows the balance of evidence the memory keeps, not the policy that kept it. The decision model adds a third failure, a form of social proof: it gives extra weight to facts attributed to many agents. Removing the attribution tags, or reassigning them at random with line order fixed, raises accuracy in the classic configuration by 13 to 24 points. In natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.

*Auto-collected on 2026-10-29*