Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game
Field: Machine Learning Authors: Muhan Gao, Hangxiao Zhu, Xinqi Su, Yu Zhang, Daniel Khashabi, Kuan-Hao Huang Published: 2026-11-10 arXiv: 2611.14487
Abstract (translated)
Research ideation is iterative: researchers refine their ideas step by step based on feedback. Yet existing evaluations obscure this search process, making it difficult to understand how LLMs explore the space of research ideas and what limits their progress. Re-Search adapts the format of Twenty Questions to research ideation: given only the research problem of a published paper, a model must rediscover the paper's core idea by asking yes-or-no questions. Built from 200 ICML 2026 papers, Re-Search tracks how a model's idea evolves as it asks questions and receives binary oracle feedback, isolating ideation from execution.
Key Findings
- Across six recent LLMs, nearly all progress happens within the first 30 of 100 questions; this early phase largely determines the outcome, and most models plateau afterward.
- Controlled interventions over the candidate question and hypothesis spaces show the bottleneck is generation, not judgment: models can recognize good ideas or questions when provided, but often cannot propose them themselves.
- Trajectory analysis reveals two characteristic failure modes: over-exploitation of a single direction, or over-exploration of too many directions.
Design Implications
These findings suggest concrete design principles for automated research systems: since models can judge candidates they cannot generate, enriching the search with ideas and questions from external sources may be more effective than improving a single model's generation capability.
---
*Auto-collected on 2026-11-12.*