The Counterintuitive Fact
Ask GPT-4 to write three different stories and it gives you three versions. Read them closely: the character names change, the locations change, but the narrative skeleton is nearly identical.
This is not a coincidence. Post-training systematically kills output diversity while improving quality. The model becomes "better," but also more "like itself." The paper quantifies this cost: on multiple semantic diversity metrics, SFT/RLHF-trained models lose an average of 20-40% compared to their base models.
The problem is that many tasks can't rely on "good" alone:
- Story generation: needs genuine narrative branching, not reskinned rewrites
- Reinforcement learning: policy exploration requires enough entropy in the action distribution, or the agent gets stuck in local optima
- Data augmentation: if LLM-generated training data all looks the same, augmentation is pointless
- Lexical: n-gram overlap, distinct-n, self-BLEU—only surface repetition
- Semantic: embedding distance, sentence similarity—repetition of meaning
- Diversity: 70.3% higher than the SFT baseline on LLM-GED, 28% higher than a multi-model ensemble baseline
- Quality: human evaluation shows parity with the post-trained model—no sacrifice
- Efficiency: only one model needed at inference, no ensembling
- Creativity mode depends on the base model existing. If you only have a post-trained model, you can't construct contrastive data—a real deployment limitation.
- LLM-GED uses an LLM for evaluation, risking circular reasoning. The authors align it with human evaluation, but the metric's reliability needs more validation.
- The RL environments are relatively simple. Whether diversity gains translate to return gains in more complex settings (multi-agent, long horizon) remains to be seen.
The core question of CreativeInstruct is simple: can a model keep post-training quality while recovering base-model creativity?
Method: A Learnable "Creativity Switch"
The idea is remarkably simple. During instruction fine-tuning, the authors insert a special token [StartCreativity] into the training data, telling the model: "for the following segment, generate in the style of the base model."
The approach has three steps:
Step 1: Collect contrastive data. For the same instruction, generate outputs from both the base model (high diversity, low quality) and the post-trained model (high quality, low diversity). This forms the two poles of a creativity-quality axis.
Step 2: Build labeled training data. Label the base model's outputs as "creativity mode" and the post-trained outputs as "quality mode," training the model to switch into creativity mode when it sees [StartCreativity]. In essence, the model learns a conditional distribution: given the "creativity signal," output like the base model; without it, output like the post-trained model.
Step 3: Switch on demand at inference. Want creativity? Insert [StartCreativity]. Want stability? Don't. One model, two modes.
This is fundamentally different from traditional temperature tuning. Temperature only resamples from an existing distribution—changing how aggressively you sample. CreativeInstruct changes the distribution itself, biasing the model toward the wider distribution of the base model. An analogy: temperature tuning is diving with different poses in the same pool; CreativeInstruct switches to a different pool.
LLM-GED: A Metric That Actually Measures Narrative Diversity
The paper's second contribution is LLM-GED (Graph Edit Distance). Existing diversity metrics fall into two categories:
But story diversity isn't just about words or sentences. Two stories can use completely different words while telling the same "hero's journey." LLM-GED uses an LLM to parse stories into "abstract narrative graphs" (nodes are narrative units, edges are relations), then compares graphs via edit distance. Larger graph differences mean greater narrative-level diversity.
This is conceptually similar to AST comparison in software engineering: don't compare code literally, compare syntax tree structure—only here the syntax tree becomes a narrative graph.
Results: No Quality Loss, Big Diversity Gains
On narrative generation tasks:
Even more interesting are the RL experiments. Using CreativeInstruct as the policy initialization for RL training significantly improves exploration efficiency—final returns are 29% higher than standard SFT initialization across several RL environments. This confirms an intuition: the exploration bottleneck in RL is fundamentally a lack of policy diversity. If the initialized policy concentrates in one region, no exploration algorithm can escape.
Why This Paper Is Worth Reading
1. It puts the "post-training kills diversity" problem on the table. Many people know SFT narrows models, but few have systematically quantified the cost. CreativeInstruct gives a concrete number: 20-40% diversity loss.
2. The [StartCreativity] design is extremely engineering-friendly. No architecture changes, no extra inference cost—just one token. This "minimally invasive" design philosophy is worth learning from.
3. LLM-GED has independent value as a metric. Evaluating LLM output diversity has always been hard, and the graph-edit-distance approach is closer to "narrative-level difference" than purely lexical or semantic metrics.
4. The RL experiments reveal a deep connection: creativity isn't just "writing prettier stories"—it is a foundational resource for exploration. A policy without creativity is "spinning in a local optimum" in RL.
An Honest Assessment
The paper isn't without issues:
The answer appears to be: yes—and it can be recovered.
---
Paper: Sahu, Bansal, Stengel-Eskin. *CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity*. https://arxiv.org/abs/2608.07460, 2026.
Code: https://github.com/ananya-sahu/CreativeInstruct