Artificial Hivemind: The Open-Ended Homogeneity of Language Models (NeurIPS 2025 Best Paper)

Forum topic · ✨步子哥 · 2025-12-03

Summary

"Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)", a NeurIPS 2025 Best Paper, studies why large language models produce homogeneous outputs in open-ended generation. The authors built Infinity-Chat, a dataset of 26K diverse real-world open-ended user queries, plus the first comprehensive taxonomy covering the full spectrum of open-ended prompts, organized into 6 top-level categories (including creative content generation and brainstorming) and 17 subcategories. The study includes 31,250 human annotations covering absolute ratings and pairwise preference choices, with each sample labeled by 25 independent annotators. Findings reveal a significant hivemind effect: individual models repeat similar answers across samples, and different models produce strikingly similar outputs. Even with diversity-enhancing decoding strategies, over 60% of responses exceed 0.8 similarity. The paper argues this creates a bland collective consensus lacking individuality and raises concerns about long-term homogenization of human thought. Authors span the University of Washington, Carnegie Mellon University, Allen Institute for AI, Lila Sciences, and Stanford University.

Overview

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) received a NeurIPS 2025 Best Paper Award. The research examines how large language models fail to generate diverse, human-style creative content, instead converging toward homogeneous outputs — an "artificial hivemind" where all models produce a bland, uniform collective consensus with no independent character.

Background

Large language models perform poorly at generating diverse, human-quality creative content; their outputs tend toward homogenization. This raises concerns that human thinking may itself homogenize over time through repeated exposure to similar AI outputs.

Methods

Infinity-Chat dataset: 26K diverse, real-world, open-ended user queries
Taxonomy: the first comprehensive taxonomy characterizing the full spectrum of open-ended prompts faced by language models, with 6 top-level categories (creative content generation, brainstorming and ideation, etc.) and 17 subcategories
Human annotation: 31,250 annotations covering both absolute ratings and pairwise preference choices; each sample was labeled by 25 independent annotators

Key Findings

The study reveals a significant artificial hivemind effect in open-ended generation:

Intra-model repetition: the same model repeatedly generates similar responses
Inter-model homogeneity: different models produce strikingly similar outputs
Even with diversity-enhancing decoding strategies, more than 60% of responses exceed 0.8 similarity

Significance

Sets a new benchmark for datasets and evaluation
Advances scientific understanding and addresses major social challenges rather than only improving technical performance
Provides key insights for mitigating long-term AI safety risks posed by the hivemind effect
Offers a baseline and direction for building more diverse AI systems aligned with the plurality of human needs

As the authors state: this work makes an important and timely contribution to understanding diversity, value pluralism, and the social impact of modern language models.

Authors and Affiliations

Authors: Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Yejin Choi

Institutions: University of Washington, Carnegie Mellon University, Allen Institute for AI, Lila Sciences, Stanford University

Venue: NeurIPS 2025