Artificial Hivemind: The Open-Ended Homogeneity of Language Models — NeurIPS 2025 Best Paper

Forum topic · ✨步子哥 · 2025-12-03

Summary

A NeurIPS 2025 best paper titled 'Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)' reveals that large language models produce strikingly homogeneous outputs in open-ended generation. The study introduces the Infinity-Chat dataset, containing 26K diverse, real-world open-ended user queries, along with the first comprehensive taxonomy of open-ended prompts spanning 6 top-level categories and 17 subcategories. The findings are backed by 31,250 human annotations covering absolute ratings and pairwise preference judgments, with 25 independent annotators per sample. The research documents both intra-model repetition (the same model generating similar answers repeatedly) and inter-model homogenization (different models producing remarkably similar outputs). Even with diversity-enhanced decoding strategies, over 60% of responses showed similarity scores above 0.8. Authored by researchers from the University of Washington, Carnegie Mellon University, Allen Institute for AI, Lila Sciences, and Stanford University, the work raises concerns about human thought homogenization through repeated exposure to similar AI outputs and offers insights for mitigating long-term AI safety risks.

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

A NeurIPS 2025 Best Paper, presented as a forum post on zhichai.net, examines why large language models fail to generate diverse, human-like creative content and instead converge toward homogeneous outputs.

Research Background

Large language models underperform at generating diverse, human-style creative content; their outputs tend toward homogenization. This raises concerns that human thinking may become homogenized over the long term through repeated exposure to similar AI outputs. The paper describes a bland, impersonal "mediocre collective consensus" — like worker bees in a hive lacking independent thought.

Methodology

Infinity-Chat dataset: 26K diverse, real-world, open-ended user queries.
Taxonomy: The first comprehensive taxonomy covering the full spectrum of open-ended prompts faced by language models, with 6 top-level categories (e.g., creative content generation, brainstorming and ideation) and 17 subcategories.
Human annotation: 31,250 human annotations covering both absolute ratings and pairwise preference choices, with 25 independent annotators per sample.

Key Findings

The study reveals a significant artificial hivemind effect in open-ended generation, characterized by:

Intra-model repetition: the same model repeatedly generates similar answers to equivalent prompts.
Inter-model homogenization: different models produce strikingly similar outputs to each other.
Even when using diversity-enhanced decoding strategies, more than 60% of responses had similarity scores above 0.8.

Significance

Sets a new standard for datasets and benchmarks.
Advances scientific understanding and addresses major societal challenges, rather than merely improving technical performance.
Provides key insights for mitigating long-term AI safety risks posed by the artificial hivemind effect.
Offers a benchmark and direction for building more diverse AI systems aligned with pluralistic human needs.

As the poster concludes: this work makes an important and timely contribution to understanding diversity, value pluralism, and the societal impact of modern language models.

Authors & Affiliations

Authors: Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Yejin Choi

Institutions: University of Washington, Carnegie Mellon University, Allen Institute for AI, Lila Sciences, Stanford University

Venue: NeurIPS 2025