Verifiable Social Reasoning for LLM Assistants: Introducing Fuse, a Multi-Agent Simulation Framework

Forum topic · 小凯 · 2026-09-17

Summary

Researchers introduce Fuse, a multi-agent simulation framework for evaluating user-mediated social reasoning in LLM assistants, presented in arXiv paper 2609.17496. In Fuse, a target agent with a hidden motive interacts with other agents, including one representing the user, who then consults the evaluated assistant to infer the target's motive—providing verifiable ground truth by construction. Simulation faithfulness was validated through a human study with 24,000 annotations. Applying Fuse to 12 LLMs, the authors systematically isolated key factors: user mediation amplifies the inherent difficulty of social reasoning; LLMs show systematic sensitivity to biased user framing; models may need more detail than humans to make correct predictions; and longer conversations do not always improve performance, even when clarification questions are possible. The framework and a 21,000-example dataset are open-sourced, enabling reproducible evaluation of social reasoning in consultation settings.

Paper Overview

Research area: NLP
Authors: Amir Taubenfeld, Zorik Gekhman, Avigail Grinstein-Dabush, Itay Laish, Ariel Goldstein, Marian Croak, Avinatan Hassidim, Yossi Matias, Amir Feder
Published: 2026-09-15
arXiv: 2609.17496

Abstract

LLM assistants are widely used for daily social advice, yet evaluating their social reasoning in such consultation settings remains challenging since (i) it requires setups where the assistant learns about social situations from subjective user narratives, and (ii) social properties, such as others' intentions, typically lack verifiable ground truth.

To address these challenges, the authors introduce Fuse, a multi-agent simulation framework for studying user-mediated social reasoning. In Fuse, a target agent with a hidden motive interacts with other agents including one representing the user, who then consults the evaluated assistant to infer the target's motive—providing verifiable ground truth by construction.

Simulation faithfulness is validated through a human study with 24,000 annotations. The framework is applied to 12 LLMs, systematically isolating key factors:

User mediation amplifies the inherent difficulty of social reasoning.
LLMs exhibit systematic sensitivity to biased user framing.
Models may require more detail than humans to reach correct predictions.
Longer conversations do not always improve performance, even when opportunities for clarifying questions are provided.

Fuse and a dataset of 21,000 examples are open-sourced for further research.

---

*Auto-collected on 2026-09-17*

Verifiable Social Reasoning for LLM Assistants: Introducing Fuse, a Multi-Agent Simulation Framework

Summary

Paper Overview

Abstract

Tags