CreativeInstruct: A Token-Level Creativity Switch for Post-Trained LLMs

Forum topic · ✨步子哥 · 2026-08-10

Summary

This paper addresses a counter-intuitive finding: standard SFT and RLHF post-training systematically reduces output diversity by 20-40% across semantic metrics, even as quality improves. The authors introduce CreativeInstruct, a lightweight method that inserts a special [StartCreativity] token into instruction tuning data so the model can conditionally switch between a quality-oriented mode (post-trained behavior) and a creative mode that mimics the broader distribution of the base model. Training uses paired outputs from base and post-trained models, letting one model serve both roles without architectural changes or extra inference cost. The paper also proposes LLM-GED, a graph edit distance metric that parses stories into abstract narrative graphs to measure structural rather than lexical diversity. Experiments show 70.3% higher LLM-GED diversity over SFT baselines, unchanged human-judged quality, and 29% higher final return when used as policy initialization for reinforcement learning, suggesting diversity is a key bottleneck for exploration.

Key points

Core problem: Post-training (SFT/RLHF) boosts output quality but systematically destroys diversity, costing 20–40% across multiple semantic diversity measures compared with the base model.
Method: Insert a special token [StartCreativity] into instruction-tuning data so a single model can switch modes on demand. Contrastive pairs are generated by sampling the same prompt from the base model (creative) and the post-trained model (quality), then training the model to behave like the base when the token is present and like the post-trained model otherwise.
Difference from temperature: Temperature only reshapes an existing distribution; CreativeInstruct changes the distribution itself, biasing generation toward the base model's wider output space.
New metric — LLM-GED: An LLM parses each story into an abstract narrative graph (nodes = narrative units, edges = relations), and diversity is measured as graph edit distance between stories. This captures structural repetition (e.g., shared "hero's journey" templates) that lexical metrics like distinct-n or self-BLEU and embedding-similarity metrics both miss.
Results on narrative generation: +70.3% LLM-GED over an SFT baseline and +28% over a multi-model ensemble baseline, with human evaluation showing quality on par with the post-trained model and no extra inference cost.
Results on RL: Using CreativeInstruct as the policy initialization for RL training yields +29% final return over standard SFT initialization on the tested environments, supporting the claim that policy-entropy/exploration is the real bottleneck.

Why it matters

1. It quantifies a cost most alignment work ignores: making models "better" can make them uniformly so. 2. The [StartCreativity] design is minimally invasive — no architecture changes, no extra models. 3. LLM-GED is an independently useful evaluation tool for narrative diversity. 4. RL results suggest creativity is a substrate for exploration, not just a storytelling property.

Honest caveats

Requires access to the base model to construct contrastive data, limiting applicability when only a post-trained checkpoint is available.
LLM-GED relies on an LLM judge, introducing circularity risk; human alignment is reported but broader validation is still needed.
RL experiments use relatively simple environments; transfer to multi-agent or long-horizon settings is unverified.

References

Paper: Sahu, Bansal, Stengel-Eskin. *CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity*. arXiv:2608.07460, 2026. https://arxiv.org/abs/2608.07460
Code: https://github.com/ananya-sahu/CreativeInstruct