onPanda: Bringing Word-Style Revision Tracking to LLM Alignment Annotation

onPanda is an open-source annotation tool from StepFun and Xiamen University that lets annotators perform token-level corrections on model outputs instead of manually rewriting or coarse preference labeling. Paper: arXiv:2609.24983, code, dataset.

The problem

• Manual rewriting produces off-policy data: the text reflects the annotator's writing style, not the model's sampling distribution, so SFT teaches mimicry rather than good generation.
• Preference labeling is efficient but gives coarse supervision — it never says *which token* went wrong, and fails entirely when the model can't sample a good response.

The locate-correct-continue loop

1. Locate: The annotator reads the response and finds the first bad token. Each token displays a color band showing generation probability (deeper red = lower probability), which doubles as a diagnostic signal. 2. Correct: Hovering reveals the model's top-20 candidate tokens at that position with probabilities; click to replace, or double-click to type arbitrary text. 3. Continue: Everything after the position is truncated, and the model regenerates from the corrected prefix.

A key benefit: no need to re-read the full response after edits, since downstream text is regenerated to be consistent with the corrected prefix.

52% time reduction, on-policy by construction

A controlled study reports a 52% median reduction in annotation time versus manual post-editing, attributed to:

• Most edits being single clicks from candidate lists rather than keyboard input
• A linear workflow with no re-reading loops
• The model generating the vast majority of tokens

Because nearly all tokens come from the model's own sampling (original rollout or continuation from corrected prefixes), the final data is on-policy — the model learns which branch points to take within its own language space.

The annotation tree

Each session is stored as a tree; each correction forks a new node. This yields:

• SFT data from nodes marked is_good=Y
• Preference pairs where positives and negatives share prefixes and diverge at the correction point — token-level one-to-one correspondence at each position
• Token-level correction triples: (negative sample, rejected token + position, selected token)

Since conversations are stored structurally with response templates applied only at render time, the same data can be continued with different models (e.g., start with Qwen, continue with Llama).

Agent trajectories

A response template mechanism converts bidirectionally between structured messages (reasoning, content, tool calls) and the model's native token stream — so even special tokens and tool-call parameters are directly visible and correctable.

External environments connect via MCP; a harness_to_mcp adapter wraps Claude Code, Codex, and OpenClaw as MCP servers. Tool calls can require annotator approval: fix parameters before execution, or reject with textual guidance (rejected trajectories are kept as negatives). Critically, corrected tool calls are actually executed — real environment feedback, enabling in-process intervention rather than post-hoc scoring.

Panda-CVL dataset and benchmark

The team releases the Panda-CVL dataset plus a token-level correction benchmark, turning "find the first bad token and its correct replacement" into a quantitatively evaluable task for future auto-correction research.

Limitations

• The user study is small; the 52% figure needs validation at larger scale and across more task types.
• Post-training methods for token-level correction data remain future work.
• Requires APIs supporting continue_final_message and logprobs (vLLM supports both; not all frameworks do). Free-edited tokens also need prompt_logprobs.
• The core assumption — that errors are locatable at a specific token — may not hold when problems stem from overall reasoning strategy, fact retrieval, or question understanding.

The bigger picture

The paper proposes an annotation-training flywheel: token-level correction data → better models → easier annotation → better data. Token-level corrections offer supervision finer than preferences and more on-policy than rewrites. Beyond a tool, onPanda defines a data format that opens a new supervision-signal space for future post-training methods.

Tags context: from StepFun + Xiamen University; open-source Python library.