When Many Small Agents Share a Whiteboard: Hallucination Amplification in Resource-Constrained Multi-Agent Vision Systems

Paper Snapshot

| Item | Detail | |------|--------| | Title | Diagnosing Failure Modes of Shared-State Collaboration in Resource-Constrained Visual Agents | | Author | Yunpeng Zhou | | Affiliation | Independent researcher | | arXiv | 2605.31354 | | Date | 2026-05-29 | | Core finding | Shared workspaces among modular visual agents with small (4B-8B) models amplify rather than suppress hallucinations. Two failure modes dominate—Noise Reinforcement and Policy Collapse. Scaling compute without explicit verification can correlate negatively with accuracy. |

The Shared Whiteboard Problem

Imagine five interns locked in a conference room with a fifty-page document and one question: does the contract amount on page three match the budget in Appendix B? Each intern can hold only a few pages of context, so they divide labor: A reads the early pages, B the middle, C the appendix, D aggregates on a whiteboard, E checks the final answer.

A misreads a decimal point and writes "one million." B notes only the first of two tables in Appendix B. C reads a subtotal, not the grand total of "eighty thousand." D stitches these into "inconsistent, gap of two hundred thousand." E accepts the synthesis. The final answer is wrong, and worse than any single intern's solo attempt: individual errors stack, amplify, and cross-validate into a self-consistent false narrative.

This is the core phenomenon the paper documents under audit.

CoSee: A Microscope for Collaboration

CoSee (Collaborative See) is an auditing framework that decomposes agent collaboration into three explicit actions:

• Read — pull existing notes from the shared workspace.
• Write — insert new information based on observations and prior notes.
• Verify — confirm the reliability of written notes.

In idealized collaboration, every Write is followed by a Verify. With small models, Verify is often skipped or shallow. CoSee makes the otherwise opaque workflow traceable step-by-step: who wrote what, who cited it, and whether anyone actually checked.

The evaluation uses Document Visual Question Answering (DocVQA) across three benchmarks representing different visual densities and structural complexities: multi-page documents, charts, and web screenshots. Each scenario demands multi-agent coverage because no single agent can see everything within a limited context window.

Failure Mode 1: Noise Reinforcement

Noise Reinforcement is the mechanism by which uncorrected errors gain authority through citation. An agent misreads "3.2%" as "32%" and writes it. The next agent cites it with elaborate justification, and the final agent, seeing the lengthy prior analysis, treats the value as peer-reviewed.

The paper finds that error rate grows super-linearly as agent count rises from 2 to 5. More writers mean more injection points without proportional verification capacity.

Two noise subtypes are distinguished:

• Perceptual noise — visual misreads driven by insufficient encoder resolution on small fonts and dense layouts.
• Reasoning noise — over-inference such as "sales rose, so profits rose," ignoring cost changes. Reasoning noise is more dangerous because it wears a logical veneer.

CoSee also surfaces an unsettling "beautification effect": the more often an error is cited, the more credible it appears—a "repetition-equals-truth" bias inside AI systems.

Failure Mode 2: Policy Collapse

Policy Collapse is subtler: as the workspace fills, agents shift from careful analysis to superficial output. The paper reports quantitative evidence. The same 4B model working solo produces an average of 47 tokens and 3.2 factual claims per answer; in collaboration, output drops to 19 tokens and 1.1 claims. The agent becomes lazy, not from intent but from cognitive overload.

Diffusion of responsibility compounds the effect, akin to the human bystander effect. A controlled experiment comparing role-assigned versus role-less configurations reveals that explicit role labels paradoxically reduce performance: agents given the title "verifier" write many "verified" tags but perform little actual cross-checking. Mere role-play prompting does not generate real verification behavior.

The Paradox: More Compute, Worse Results

The most counter-intuitive finding: scaling compute without explicit verification can correlate negatively with accuracy.

| Setup | Rounds | Total Compute (GFLOPs) | Accuracy | |-------|--------|------------------------|----------| | No verification | 2 | 12.4 | 61% | | No verification | 3 | 18.6 | 58% | | No verification | 5 | 31.0 | 53% | | Explicit verification | 2 | 15.8 | 64% | | Explicit verification | 3 | 23.7 | 71% | | Explicit verification | 5 | 39.5 | 78% |

The Pareto frontier under no-verification bends downward after a threshold; compute accumulates but accuracy falls. Two mechanisms drive the degradation:

1. Historical pollution — each round injects 15-20% erroneous notes that never get cleaned; by round 5, roughly 40% of workspace content contains detectable errors. 2. Reasoning-chain bloat — longer chains overwhelm the small models' long-range dependency handling, scattering attention onto irrelevant details.

The Bottleneck Is the Channel, Not the Brain

The conclusion: for resource-constrained agents, the bottleneck lies in communication fidelity, not reasoning depth.

Communication fidelity has four layers:

• Information fidelity — is what is written accurate?
• Structural fidelity — are parent-child relations preserved?
• Verification fidelity — has it been cross-checked?
• Priority fidelity — is critical content highlighted and noise filtered?

Small models lose on all four. The workspace devolves from collaboration tool into rumor mill. For enterprises building small-model "AI teams" to cut costs, the warning is direct: without investment in communication integrity, five cheap agents can perform worse than one capable model.

Suggested mitigation directions (not deeply evaluated in the paper):

• Structured shared workspaces with templated notes carrying source, confidence, and verification tags.
• Redundant verification requiring at least two independent agents to confirm any critical note.
• Information decay mechanisms introducing half-lives so unused notes are downgraded or removed, bounding workspace growth.

Limitations

• Single-author scope. Only 4B-8B models tested; behavior at 14B+ remains unknown, and larger models may produce more confident errors in a trade-off.
• Task-domain limits. DocVQA has deterministic answers. Open-ended tasks (creative writing, strategy debate) lack clear verification criteria.
• Verification cost. Explicit verification adds roughly 25-30% compute overhead, which may not suit edge or real-time deployments.
• No human-team comparison. Noise Reinforcement and Policy Collapse resemble known group pathologies (echo chambers, diffusion of responsibility). The paper does not disentangle AI-specific from cooperation-general causes.
• No ablation between failure modes. The interaction between Noise Reinforcement and Policy Collapse—for example, whether lazy agents verify less—is unexplored.
• Limited remedies. Diagnosis is thorough; prescribed treatments are sketched rather than validated.

References

1. Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. *ICLR 2023*. arXiv:2210.03629. 2. Shinn, N., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. *NeurIPS 2023*. arXiv:2303.11366. 3. Li, Y., et al. (2023). Evaluating Object Hallucination in Large Vision-Language Models. *EMNLP 2023*, 292–305. arXiv:2305.10355. 4. Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. *NeurIPS 2022*. arXiv:2201.11903. 5. Kahneman, D. (2011). *Thinking, Fast and Slow*. Farrar, Straus and Giroux.