onPanda: Bringing Revision Tracking into LLM Annotation — Half the Time, On-Policy Distribution Intact

An Annotator's Morning

Imagine you are an RLHF annotator. The model produces an answer, you read it, and from the third sentence it starts drifting off track — one wrong word cascades through the rest of the reasoning.

Your options are limited:

• Manual rewriting: delete everything from the bad sentence onward and write it yourself. But your prose is not what the model would say — it carries your style and rhythm. Training on this data with SFT teaches the model to imitate your writing, not to generate good answers. This is the classic off-policy problem: training data diverges from the model's own sampling distribution.
• Preference labeling: score two responses. Cheap, but the supervision signal is coarse — you say which is better without indicating *which token* the response went wrong at. And if the model can't sample a good response at all, what do you score?

onPanda's answer is almost plain: like Word's revision tracking, find the first wrong token, fix it, and let the model regenerate from that position.

Token-Level Correction: the locate-correct-continue loop

1. Locate: read the response and find the first unsuitable token. Each token displays a colored band — deeper green means higher generation probability, deeper red means lower. Low-probability tokens often mark where the model is "unsure." 2. Correct: hover over the token to see the model's top-20 candidate tokens and their probabilities. Click to swap; if nothing fits, double-click and type any text. 3. Continue: everything after that position is truncated, and the model regenerates from the corrected prefix as the assistant message's starting point.

Then loop — read, find, fix, continue — until the response is satisfactory.

An underrated benefit: you never need to re-read the whole response after fixing it. In manual rewriting, fixing sentence three may break sentence seven. onPanda's truncate-and-continue mechanism eliminates this — sentence seven is regenerated by the model itself, so it is naturally coherent with the corrected prefix.

52% Time Reduction — and What It Means

The paper reports a 52% median reduction in annotation time compared to manual post-editing. The reduction comes from three sources:

• Most corrections are mouse clicks, not typing. Picking from top-20 candidates is an order of magnitude faster than hand-writing text.
• A linear workflow. Read top to bottom, fix as you go, no backtracking — eliminating the repeated full re-reads of traditional annotation.
• The model generates most tokens itself. The annotator touches only a few key positions.

That last point is the heart of the design. Since nearly all tokens come from the model (sampled from the original prompt or continued from the corrected prefix), the final response lies almost entirely within the model's sampling distribution — that is what on-policy means. The data is "what the model would actually say," merely nudged by a human at critical forks.

SFT on this data teaches the model which forks to take in its own language space, not how to mimic human essays. For preference learning, positive and negative samples pair naturally: they share the prefix before the correction point and diverge exactly at it, yielding precisely located, directionally clear training signals.

The Annotation Tree: Every Intermediate Version Is an Asset

onPanda organizes each session as an annotation tree. Each node is a complete conversation state (messages, tools, annotations). Every correction forks a new node, with the parent being the pre-correction state.

• SFT data: nodes marked is_good=Y export directly as SFT samples.
• Preference data: positive/negative nodes under the same prompt are paired; negatives are usually ancestors of positives, sharing prefixes before the correction point. A key property: tokens correspond one-to-one at every position, giving naturally balanced optimization signals.
• Token-level correction data: expressible as a triple (negative sample, rejected token and its position, chosen token).

Because conversations are stored in structured form and response templates are applied only at rendering and correction time, the same data can be continued and corrected by different models — you can annotate half with Qwen and switch to Llama, provided the response templates match.

Agent Trajectories: Correcting Tool Calls Too

Modern reasoning models and agents emit structured messages — reasoning, content, tool calls — which cannot be directly token-corrected. onPanda's response template mechanism converts bidirectionally between structured messages and the model's native token stream: rendering expands structured messages into sequences with special tokens (e.g. </think>, |_begin|) for display and correction; parsing restores the generation stream to structured form for storage and tool execution.

Special tokens can be directly seen and corrected — annotators can edit reasoning chains and tool call arguments under one unified interface.

External environments connect via MCP. A harness_to_mcp adapter wraps existing harnesses — Claude Code, Codex, OpenClaw — as MCP servers. Tool calls can require annotator approval before execution: fix bad arguments first, or reject the call with textual guidance, automatically leaving the rejected trajectory as a negative sample.

The key advance: agent trajectories arise from multi-step environment interaction, so any correction must actually execute tools and get real feedback before generation continues. onPanda closes that loop — this is not retrospective scoring, but in-process intervention.

Panda-CVL Dataset and the Token-Level Correction Benchmark

The paper releases the Panda-CVL dataset, annotated with onPanda, plus a token-level correction benchmark. This turns "find the first unsuitable token and give the correct replacement" into a quantitatively evaluable task, giving automatic correction research a unified evaluation standard.

Honest Assessment: The Boundaries

• The user study is small ("a small controlled study"); the 52% figure needs validation at larger scale and across more task types.
• Post-training methods for token-level correction data remain future work. The data formats and pairing schemes exist, but how much they improve trained models is unresolved.
• API requirements are nontrivial: continue_final_message (continue from prefix) and logprobs (top-k candidates). vLLM supports both; not every inference framework does out of the box.
• Free-typed tokens lack probability info initially, requiring an extra prompt_logprobs request.

A deeper question: how often does the "first unsuitable token" assumption hold? When a model misunderstands the question from the start or fails at overall reasoning strategy or factual retrieval, is there a single token to fix? onPanda's interaction design implicitly assumes problems are locatable at token granularity — how broadly that holds is not deeply discussed.

The Bigger Picture: an Annotation-Training Flywheel

The paper's future work sketches an annotation-training flywheel: token-level correction data improves post-training → the model improves → less/easier annotation needed → annotation efficiency rises further → the model improves again...

The flywheel hinges on token-level correction providing supervision far finer-grained than preferences and more on-policy than rewriting. If post-training objectives are designed to exploit positional and directional information — beyond plain SFT and DPO — the flywheel could spin faster than expected.

This is onPanda's most imaginative contribution: not just a tool, but a data format definition. Precisely recording "which token a human changed, and how" opens a new supervision signal space.

Conclusion

onPanda looks simple — Word's revision tracking, moved into LLM annotation. But that move resolves a set of subtle tensions: on-policy data vs. human intervention, coarse preferences vs. fine supervision, annotation efficiency vs. data quality. The 52% time cut is the visible gain; on-policy distribution preservation is the invisible but more important one; the token-level correction data format is the future-facing one.

For teams working on RLHF or agent alignment, onPanda is worth a serious look. It is open source, connects via MCP, and supports Claude Code/Codex/OpenClaw — the barrier is low enough to test the locate-correct-continue loop yourself.

---

Paper: onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction arXiv: https://arxiv.org/abs/2609.24983 Code: https://github.com/on-panda/on-panda-python Dataset: https://on-panda.github.io/research Authors: StepFun + Xiamen University