ATPO: Safety Is Not a Switch but a Knob — Controllable Precision-Recall for Multi-Label Video Safety Detection
Paper Overview
Research area: Video Safety / RLHF Authors: Guangyu Yang, Haoze Yang, et al. Institutions: Xiaohongshu (RED) / Shanghai Jiao Tong University arXiv: 2610.02019 Code: bruceyg.github.io/ATPO-project-page
---
Can a Video Be "Unsafe" in More Than One Way?
Imagine you are moderating a short video. In the frame, two people are fighting (violence); the background music contains insulting lyrics (hate speech); and in the comments, someone is posting harassing messages (harassment).
Is this video "unsafe"? Of course. But is it unsafe in a single binary "unsafe = yes" sense? No. It involves three categories at once: violence, hate speech, and harassment. If you only answer "unsafe," you lose a huge amount of information — moderators won't know whether to delete the video, the music, or the comments.
This is the first problem with existing video safety detection systems: they reduce safety detection to binary classification (safe/unsafe), ignoring the multi-label nature of unsafe content.
The second problem is subtler: different moderation scenarios require different precision-recall trade-offs.
- Human review teams: prefer over-flagging (high recall), letting moderators filter false positives
- Automated takedown systems: prefer missing some (high precision), avoiding wrongful removal of legitimate content
- Different categories (violence vs. pornography) carry different legal thresholds
- Large α → penalize FP more → more conservative (high precision)
- Large β → penalize FN more → more aggressive (high recall)
- Baseline Jaccard Index: 40.66
- ATPO Jaccard Index: 75.44
- Wrongfully deleting legitimate content (FP) → user complaints, public backlash, brand damage
- Missing harmful content (FN) → regulatory penalties, legal risk, platform reputation
But existing models have fixed training objectives — you either optimize accuracy or F1, with no way to switch flexibly between scenarios.
ATPO's Two Innovations
ATPO (Adaptive Tversky Policy Optimization), proposed by Xiaohongshu together with Shanghai Jiao Tong University, uses reinforcement learning to solve both problems.
Innovation one: Multi-Label Video Safety Detection (Multi-VSD)
Instead of binary classification, the model simultaneously predicts multiple unsafe categories. Its output is a set of labels (violence: 0.8, hate: 0.6, harassment: 0.3) rather than a single "unsafe" score.
Innovation two: Adaptive Tversky Reward (ATR)
This is the core innovation. The Tversky Index, proposed by Tversky in 1977, has two parameters α and β that control the penalty weights on false positives (FP) and false negatives (FN):
ATPO's approach: dynamically adjust α and β during training. Rather than fixing one target, the model learns to perform well under different α/β settings. After training, deployment only requires specifying α/β to "drive" the precision-recall curve to any operating point.
The Numbers
On the SafeWatch-Bench-Real dataset:
The Jaccard Index is the standard metric for multi-label tasks (intersection over union). Going from 40.66 to 75.44 is close to doubling — a remarkable gain in the already crowded field of safety detection.
More important is the controllability experiment: ATPO reliably adjusts its precision-recall operating point under different α/β settings, while baselines cannot. This means a single model can serve different moderation policies — dial up recall for human-review scenarios, dial up precision for automated takedowns.
A Deeper Insight
ATPO's design exposes a widely overlooked problem: rigid evaluation metrics limit a model's practical usefulness.
Traditional training uses F1 or accuracy as the objective, implicitly assuming "false positives and false negatives cost the same." In real content moderation, that assumption is simply false:
ATPO's Tversky framework explicitly models this asymmetry in the training process. This isn't a technical detail — it's a paradigm shift: from "optimizing a fixed objective" to "learning a controllable policy space of objectives."
A Cross-Domain Analogy
ATPO's approach reminds me of a car's adaptive suspension. Traditional suspension is fixed — either stiff (good handling but bumpy) or soft (comfortable but body roll). Adaptive suspension lets you switch modes with the push of a button.
ATPO does the same thing: instead of training a "high-precision" or "high-recall" model, it trains an "adjustable" model that can be switched on demand at deployment. The engineering value is obvious — you don't need a separate model for every moderation scenario.
Conceptual Genealogy: Another Case of Scalar Illusion
This is another instance of "scalar illusion": don't ask "is this model safe?" — ask "safe under what precision-recall trade-off, for which categories, in which scenarios?"
Safety is not a scalar; it's a multi-dimensional space. Compressing a multi-dimensional space into a scalar is like compressing a map into a single distance number — you get an "answer," but you lose all the useful information.
ATPO's contribution isn't just "improving the Jaccard Index"; more importantly, it liberates safety detection from scalar judgment into a multi-dimensional, controllable policy space.
Honest Assessment
ATPO has limitations. First, the Tversky parameters still require manual selection — the paper doesn't discuss how to automatically determine the optimal α/β. Second, multi-label training needs multi-label annotations, which cost far more than binary labeling. Finally, the experiments focus on video safety; whether the ATPO framework transfers to other multi-label tasks (medical diagnosis, sentiment analysis) remains to be validated.
But as a work that pushes content moderation from binary classification toward controllable multi-label detection, ATPO's direction is right: safety is not a switch, it's a knob.
---
Paper: arxiv.org/abs/2610.02019 Code and models: bruceyg.github.io/ATPO-project-page