onPanda: Efficient Token-Level Annotation of On-Policy Alignment Data for LLMs

Forum topic · 小凯 · 2026-09-23

Summary

onPanda is an interactive annotation tool for efficiently creating LLM alignment data and agent trajectories, presented in an arXiv paper (2609.24983) by researchers including Lei Yang, Mengyin Liu, and Xiangyu Zhang. Its core interaction is token-level correction: while reading a model response, the annotator locates the first inappropriate token, chooses a substitute from the model's candidate tokens or types corrected text, and the system truncates everything after that point and continues generation from the corrected prefix. This locate-correct-continue loop repeats until a satisfactory response is produced. A small controlled study found onPanda reduces median annotation time by 52% compared to manual post-editing. Because most tokens in the final response are generated by the model itself, the resulting data closely preserves the model's sampling distribution, making it well suited for on-policy SFT and preference data. Token-level corrections also provide fine-grained supervision signals with precise positions and naturally paired good/bad samples. onPanda can additionally connect to external tools and runtime environments, supporting interactive trajectory annotation in real environments. The authors also release the Panda-CVL dataset and a token-level correction benchmark annotated with onPanda.

Paper Overview

Field: NLP Authors: Lei Yang, Mengyin Liu, Jia Wang, Hangyu Guo, Liang Zhao, Zheng Ge, Kang An, Binxing Jiao, Qi Han, Daxin Jiang, Siqi Shen, Xiangyu Zhang arXiv: 2609.24983

Abstract

We present onPanda, an interactive tool for efficiently annotating LLM alignment data and agent trajectories. onPanda adopts token-level correction as its core interaction: while reading a model response, the annotator locates the first inappropriate token and either picks a substitute from the model's candidate tokens or types the correct text via free-form editing. The system then truncates everything after that position and continues generation from the corrected prefix, repeating this locate-correct-continue loop until a satisfactory response is obtained.

This mechanism lets annotators precisely steer model outputs at low cost: a small controlled study suggests that onPanda reduces median annotation time by 52% over manual post-editing.

Since the vast majority of tokens in the final response are generated by the model itself, the resulting data largely preserves the model's sampling distribution, making it ideal for constructing on-policy SFT and preference data. Additionally, token-level corrections recorded during annotation provide fine-grained supervision signals with precise positions, as well as naturally paired good/bad samples.

onPanda can also connect to external tools and runtime environments, supporting interactive trajectory annotation in real-world settings.

Additional Resources

The authors also release the Panda-CVL dataset and a token-level correction benchmark annotated using onPanda.

--- *Auto-collected on 2026-09-23*