Have you ever asked ChatGPT for 10 creative ideas, only to find that 7 of them are essentially the same thing rephrased?
This isn't an isolated incident. Current large language models have a systematic flaw when generating research ideas: they optimize for the quality of a single idea, not the diversity of a set. Models repeatedly produce the "safest" high-quality idea, resulting in a basket of suggestions that all look good but are highly similar to each other.
A team from Salesforce AI Research and New York University decided to tackle this problem head-on. Their paper, *Training Reasoning Models to Generate Diverse Scientific Ideas*, proposes a deceptively simple but surprisingly effective solution: use set-level reinforcement learning to train the model to actively "avoid" ideas it has already generated when producing each new one.
The result: an 8B-parameter model (based on Qwen3-8B) beat Claude Opus 4.8 and GPT-6 Astra on creative utility—a combined measure of quality and diversity.
Set-Level RL: Generate a Group at Once, Score the Whole Set
Traditional training works like this: give the model a paper, have it generate one idea, then score that idea. The model learns "how to generate good ideas."
But this paper takes a completely different approach. In a single generation, the model must write K structured "idea cards" simultaneously, each based on an input paper. The entire set is then scored jointly—each idea's quality multiplied by its distance from previous ideas.
The design is clever. Suppose you ask the model for 5 ideas:
- Idea 1: quality 8, distance = 0 (first idea, no reference)
- Idea 2: quality 8, but very similar to Idea 1, distance = 0.2 → contribution = 8 × 0.2 = 1.6
- Idea 3: quality 7, but a completely different direction, distance = 0.9 → contribution = 7 × 0.9 = 6.3
- 8B model > Claude Opus 4.8 > GPT-6 Astra (on creative utility)
- In a pilot human study, four domain experts judged the 8B model's idea sets to be as promising as Claude Opus 4.8's, with more unique ideas
- When a coding agent ran proof-of-concept experiments on these ideas, the 8B model's success@k (probability that at least one of the top k ideas succeeds) was comparable to Claude Opus 4.8
- Research topics: not one "best project," but a set of "mutually complementary directions"
- Product design: not one "best feature," but a set of "features covering different user needs"
- Strategic planning: not one "optimal strategy," but a set of "backup plans for different scenarios"
The model quickly learns that generating a third "safe but redundant" 8-point idea is worse than a "slightly weaker but distinctive" 7-point idea. It's like a band composing an album—if the first two tracks are rock, a folk song as the third makes the whole album more valuable, even if it's slightly weaker on its own.
The Two-Step Training Method
The implementation has two stages:
Step 1: SFT (supervised fine-tuning). Supervised learning on well-formed idea sets teaches the model "how to write a set of structured idea cards." This step solves format and basic capability issues.
Step 2: GRPO (set-level reinforcement learning). GRPO training with a set-level objective function. The reward function is "creative utility"—each idea's quality multiplied by its distance from previous ideas, summed up. This step solves "how to make ideas avoid each other."
The key innovation: GRPO's reward is given not to a single output but to a group of outputs. When generating the K-th idea, the model must "see" the previous K-1 ideas and actively choose a different direction. This is fundamentally different from traditional single-sample RL.
Experimental Results: A Small Model's Big Win
On the team's contamination-free benchmark:
That last point is especially important: it shows that increasing diversity did not sacrifice executability. The "different direction" ideas the model generated weren't wild fantasies—they were concepts that could actually run.
Why Does This Matter?
This paper touches a fundamental tension in AI creative generation: quality and diversity are naturally opposed.
When you train a model with RLHF to produce "good" content, it converges to a kind of "safest good"—repeatedly outputting the same type of high-quality but homogeneous content. It's like a chef who only makes one signature dish: delicious every time, but it's the only thing on the menu.
Set-level RL breaks this convergence. By explicitly adding "distance from previous ideas" to the reward function, the model is pushed to explore new directions. This resembles "adaptive radiation" in evolutionary biology—when competition is intense, species are pushed into different ecological niches.
A Deeper Implication
This paper leads me to a bigger question: is the way we evaluate AI too one-dimensional?
Current evaluation systems—whether leaderboards or benchmarks—almost all measure "the quality of a single output." But in the real world, what we often need is not "one best answer" but "a diverse and complementary set of options":
An 8B model beating Claude Opus 4.8 is also yet another proof that in AI, the design of the training method matters more than model scale. A small model with the right training objective can beat a rival tens of times larger on specific dimensions.
---
Paper: https://arxiv.org/abs/2610.22324 Code: https://github.com/ManyaWadhwa/diverge_ideation