English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game (arXiv 2611.14487)

Forum topic · 小凯 · 2026-11-12

Summary

Re-Search is a benchmark that adapts the game of Twenty Questions to research ideation in order to study how large language models explore the space of research ideas. Given only the research problem of a published paper, a model must rediscover the paper's core idea by asking yes-or-no questions and receiving binary oracle feedback. Built from 200 ICML 2026 papers, the benchmark isolates ideation from execution by tracking how a model's hypothesis evolves over up to 100 questions. Across six recent LLMs, nearly all progress occurs within the first 30 questions, after which most models plateau. Controlled interventions over the space of candidate questions and hypotheses reveal that the bottleneck lies in generation rather than judgment: models can recognize good ideas or questions when presented with them, but often fail to propose them on their own. Analysis of ideation trajectories shows failures manifest as either over-exploitation of a single direction or over-exploration of too many directions. The authors suggest that automated research systems may benefit more from enriching the search with externally sourced ideas and questions than from improving a single model's generation ability.

Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game

Field: Machine Learning Authors: Muhan Gao, Hangxiao Zhu, Xinqi Su, Yu Zhang, Daniel Khashabi, Kuan-Hao Huang Published: 2026-11-10 arXiv: 2611.14487

Abstract (translated)

Research ideation is iterative: researchers refine their ideas step by step based on feedback. Yet existing evaluations obscure this search process, making it difficult to understand how LLMs explore the space of research ideas and what limits their progress. Re-Search adapts the format of Twenty Questions to research ideation: given only the research problem of a published paper, a model must rediscover the paper's core idea by asking yes-or-no questions. Built from 200 ICML 2026 papers, Re-Search tracks how a model's idea evolves as it asks questions and receives binary oracle feedback, isolating ideation from execution.

Key Findings

  • Across six recent LLMs, nearly all progress happens within the first 30 of 100 questions; this early phase largely determines the outcome, and most models plateau afterward.
  • Controlled interventions over the candidate question and hypothesis spaces show the bottleneck is generation, not judgment: models can recognize good ideas or questions when provided, but often cannot propose them themselves.
  • Trajectory analysis reveals two characteristic failure modes: over-exploitation of a single direction, or over-exploration of too many directions.

Design Implications

These findings suggest concrete design principles for automated research systems: since models can judge candidates they cannot generate, enriching the search with ideas and questions from external sources may be more effective than improving a single model's generation capability.

---

*Auto-collected on 2026-11-12.*

Tags

#llm#research-ideation#benchmark#twenty-questions#machine-learning#arxiv#icml#automated-research

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637074