English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

CreativeInstruct: A Creativity Switch for LLMs That Stops Post-Training from Killing Diversity

Forum topic · ✨步子哥 · 2026-08-10

Summary

CreativeInstruct is a post-training method that lets large language models retain alignment-era output quality while recovering the diversity of their base models. The core idea is simple: insert a special token, [StartCreativity], into instruction-tuning data so the model learns to switch between a high-quality, low-diversity mode and a high-diversity, base-model-like mode at inference time. The paper quantifies the diversity cost of SFT/RLHF at 20-40% across semantic diversity metrics, and introduces LLM-GED, a graph-edit-distance-based metric that compares abstract narrative structures rather than surface wording. Experiments show CreativeInstruct outperforms SFT baselines by 70.3% on LLM-GED and model ensembles by 28%, with no quality loss in human evaluation, and improves RL exploration—yielding 29% higher final returns when used as policy initialization. Code and the paper (arXiv:2608.07460) are publicly available.

The Counterintuitive Fact

Ask GPT-4 to write three different stories and it gives you three versions. Read them closely: the character names change, the locations change, but the narrative skeleton is nearly identical.

This is not a coincidence. Post-training systematically kills output diversity while improving quality. The model becomes "better," but also more "like itself." The paper quantifies this cost: on multiple semantic diversity metrics, SFT/RLHF-trained models lose an average of 20-40% compared to their base models.

The problem is that many tasks can't rely on "good" alone:

  • Story generation: needs genuine narrative branching, not reskinned rewrites
  • Reinforcement learning: policy exploration requires enough entropy in the action distribution, or the agent gets stuck in local optima
  • Data augmentation: if LLM-generated training data all looks the same, augmentation is pointless
  • The core question of CreativeInstruct is simple: can a model keep post-training quality while recovering base-model creativity?

    Method: A Learnable "Creativity Switch"

    The idea is remarkably simple. During instruction fine-tuning, the authors insert a special token [StartCreativity] into the training data, telling the model: "for the following segment, generate in the style of the base model."

    The approach has three steps:

    Step 1: Collect contrastive data. For the same instruction, generate outputs from both the base model (high diversity, low quality) and the post-trained model (high quality, low diversity). This forms the two poles of a creativity-quality axis.

    Step 2: Build labeled training data. Label the base model's outputs as "creativity mode" and the post-trained outputs as "quality mode," training the model to switch into creativity mode when it sees [StartCreativity]. In essence, the model learns a conditional distribution: given the "creativity signal," output like the base model; without it, output like the post-trained model.

    Step 3: Switch on demand at inference. Want creativity? Insert [StartCreativity]. Want stability? Don't. One model, two modes.

    This is fundamentally different from traditional temperature tuning. Temperature only resamples from an existing distribution—changing how aggressively you sample. CreativeInstruct changes the distribution itself, biasing the model toward the wider distribution of the base model. An analogy: temperature tuning is diving with different poses in the same pool; CreativeInstruct switches to a different pool.

    LLM-GED: A Metric That Actually Measures Narrative Diversity

    The paper's second contribution is LLM-GED (Graph Edit Distance). Existing diversity metrics fall into two categories:

  • Lexical: n-gram overlap, distinct-n, self-BLEU—only surface repetition
  • Semantic: embedding distance, sentence similarity—repetition of meaning
  • But story diversity isn't just about words or sentences. Two stories can use completely different words while telling the same "hero's journey." LLM-GED uses an LLM to parse stories into "abstract narrative graphs" (nodes are narrative units, edges are relations), then compares graphs via edit distance. Larger graph differences mean greater narrative-level diversity.

    This is conceptually similar to AST comparison in software engineering: don't compare code literally, compare syntax tree structure—only here the syntax tree becomes a narrative graph.

    Results: No Quality Loss, Big Diversity Gains

    On narrative generation tasks:

  • Diversity: 70.3% higher than the SFT baseline on LLM-GED, 28% higher than a multi-model ensemble baseline
  • Quality: human evaluation shows parity with the post-trained model—no sacrifice
  • Efficiency: only one model needed at inference, no ensembling
  • Even more interesting are the RL experiments. Using CreativeInstruct as the policy initialization for RL training significantly improves exploration efficiency—final returns are 29% higher than standard SFT initialization across several RL environments. This confirms an intuition: the exploration bottleneck in RL is fundamentally a lack of policy diversity. If the initialized policy concentrates in one region, no exploration algorithm can escape.

    Why This Paper Is Worth Reading

    1. It puts the "post-training kills diversity" problem on the table. Many people know SFT narrows models, but few have systematically quantified the cost. CreativeInstruct gives a concrete number: 20-40% diversity loss.

    2. The [StartCreativity] design is extremely engineering-friendly. No architecture changes, no extra inference cost—just one token. This "minimally invasive" design philosophy is worth learning from.

    3. LLM-GED has independent value as a metric. Evaluating LLM output diversity has always been hard, and the graph-edit-distance approach is closer to "narrative-level difference" than purely lexical or semantic metrics.

    4. The RL experiments reveal a deep connection: creativity isn't just "writing prettier stories"—it is a foundational resource for exploration. A policy without creativity is "spinning in a local optimum" in RL.

    An Honest Assessment

    The paper isn't without issues:

  • Creativity mode depends on the base model existing. If you only have a post-trained model, you can't construct contrastive data—a real deployment limitation.
  • LLM-GED uses an LLM for evaluation, risking circular reasoning. The authors align it with human evaluation, but the metric's reliability needs more validation.
  • The RL environments are relatively simple. Whether diversity gains translate to return gains in more complex settings (multi-agent, long horizon) remains to be seen.
Overall, though, this is a rare recent work in post-training that directly attacks the diversity problem. Most papers study how to make models more accurate, more aligned, more safe. This paper asks: are we losing something important in the alignment process?

The answer appears to be: yes—and it can be recovered.

---

Paper: Sahu, Bansal, Stengel-Eskin. *CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity*. https://arxiv.org/abs/2608.07460, 2026.

Code: https://github.com/ananya-sahu/CreativeInstruct

Tags

#llm#post-training#diversity#creativity#reinforcement-learning#narrative-generation#evaluation-metrics#sft

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633310