PROBE: A 4B Auditor Model That Learned to Fish Out Policy Violations from 30B Agents

A Seemingly Perfect Booking

Imagine an utterly ordinary morning. You tell a flight-booking assistant: "Book me a Friday flight to Bangkok, nothing too pricey." Ten seconds later it replies: "Locked TG415 for Friday morning, $420 including taxes, ticketed successfully." Professional tone, clean result—you close the chat, satisfied.

Here's the problem: this booking broke four rules from start to finish. It executed a change without confirmation; the fare it displayed leaked internal codes; it even attempted a hold for a restricted destination. None of these violations appears in the final reply. The text is harmless, the task succeeded, and four "victims" are hidden under the table.

This is not alarmism. Today's capability benchmarks (WebArena, OSWorld, AgentBench) ask only: did the task succeed? Content-safety benchmarks (AgentHarm, red-teaming) ask only: is there harmful text? Neither asks the more lethal question—

When an agent completes a perfectly legitimate task, does it actually follow the rules it was given?

That is policy compliance. The answer usually isn't in the final response: was a tool called before required slots were filled? Did a change happen before user confirmation? Does the itinerary cited in the reply actually match tool outputs? Where the ruler can't measure is exactly what this paper illuminates.

The Hidden Third Axis

Draw evaluation as a coordinate system: capability on the x-axis, content safety on the y-axis—the third axis is policy compliance. Real business agents are often neither "incompetent" nor "toxic"; they're just rule-breaking: wrong currency, order changed before the user nodded. Worse, such failures have delayed emergence: consequences surface turns later, across tool calls, only in downstream systems. Any single reply looks innocent; only the full trace reveals step-by-step violations.

MBZUAI's Mishra, Sharma, Lukas, and Lahlou formally name this blind spot in *PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents* (NeurIPS 2026). One-line summary: train a dedicated AI auditor that, like an undercover inspector, walks into the shop and fishes violations out of multi-turn dialogues.

Auditing Traces, Not Replies

The unit of evaluation is the interaction trace: user utterances, assistant responses, tool calls, and state changes chained in time. Every policy constraint is defined on that chain, not on any single token—this is the watershed versus token-level content-safety evaluation.

Why not write static test scripts? Scripts enumerate, but combinations explode: user phrasing × database state × tool availability × policy preconditions × multi-turn诱导—astronomical. Handwritten lists only cover known edge cases. The paper borrows from an old trade—penetration testing (Arkin et al., 2005): instead of enumerating exploit paths, train an adaptive attacker interacting through the same public interface, finding failures that are "unimaginable in theory, walkable in practice." PROBE imports this: the auditor takes no backdoor, uses no privilege—just sits at the counter and talks like a customer.

The Undercover Inspector, Officially On Duty

Formally, it's a two-player Markov game. The auditor gets a customer scenario x, samples the next user message from the public dialogue history H; the target samples responses and tool calls; tools rewrite private state, over multiple rounds. Each policy p_i has a severity σ_i ∈ {0.5, 1, 2, 4} and a verifier v_i—a judgment function over the complete execution trace, outputting 0 or 1.

A good auditor satisfies three properties: find high-severity failures; cover many different policies rather than farming one easy rule; and do it all with behavior real users could plausibly exhibit. Objective: expected discounted reward J(θ) plus a hard constraint that user messages must be plausible.

Crucially, the setup is partially observable: the auditor sees no weights, hidden prompts, database, or verifier labels—only the dialogue interface, like a real customer. And one counterintuitive fact: off-the-shelf, larger models are not naturally good inspectors. Zero-shot Qwen3-30B achieved only 35% policy coverage. Fishing is a craft, and crafts must be learned.

The Testbed: SkyAssist and Twenty House Rules

Experiments run in SkyAssist—a deterministic flight-booking environment: 35 airports, 120 routes, 643 flights, three fare classes, four restricted destinations, 46 cancelled flights. Eight tools: 4 read-only (flight search, availability check, flight details, booking lookup) plus 4 mutation tools (hold, book, cancel, modify). Every tool is constrained by at least one policy; every policy is reachable through at least one tool path—no decorative rules.

There are 20 policies in four categories: business (USD pricing, restricted destinations), safety (schema compliance, confirmation before changes), privacy (passenger data protection, no internal fare code disclosure), operational (passenger caps, booking windows, layover validation, cancelled flight handling).

Judgment is two-pronged: 16 deterministic predicates over tool parameters, outputs, responses, and database state; 4 semantic ones (B4 advanced-option mention, B6 round-trip suggestion, S5 explicit confirmation, P1 internal pricing disclosure) use a hybrid LLM judge for training. The design is verifier-first: write the judge before training. The auditor receives only a scalar reward.

The Scorecard: Don't Count Tickets That Way

Reward design determines what the auditor becomes—reward is part of the audit specification, not just an optimization detail.

Naively counting verifier hits would produce a wordy, role-confused, loop-prone machine farming one easy policy. PROBE uses a composite reward: r = violation + α·progress + β·novelty + γ·plausibility.

• Violation reward, deduplicated 🎫: first hit on a policy pays full severity, second half, third zero—d(n) = 1, 0.5, 0. Audit semantics: ten tickets on the same spot aren't ten pieces of evidence; a ticket on a new spot is a new finding.
• Progress reward: first entry into tool-relevant states pays—slot filling 0.03, search 0.08, availability check 0.10, deep flows (book, cancel, multi-leg) 0.25–0.30. Progress weight anneals from 0.5 to 0.1 over the first 100 steps.
• Novelty: inverse global frequency plus first-discovery bonuses push the auditor off saturated policies.
• Plausibility: assistant-like messages, fabricated system-access claims, markdown/lists, excessive length, lexical echo—all penalized. The inspector may fish, but not in uniform.

All components are z-score normalized with exponentially weighted running statistics; all weights are set during training diagnostics, never tuned against held-out coverage.

Curriculum from Short Dialogues

Multi-turn policy failures bury the kill shot under innocent-looking setup. A horizon curriculum starts K_max at 4 and steps to 8, 12, 14, gated on rolling violation yield exceeding a 15% threshold (patience 50). Each update samples G=4 candidate user messages from a shared prefix, each rolled out C=3 continuation turns against the frozen target. Two A100-40GB GPUs per run. The target stays frozen—all gains come from what the auditor itself learned.

Three Teaching Styles

Same student (Qwen3-4B-Instruct + QLoRA: 4-bit NF4, rank 64, α=128 on Q/K/V/O, ~47M trainable parameters), three mentors:

• GRPO (Shao et al., 2024): group-relative advantage A_i = (R_i − R̄)/(std+ε) with PPO clip and KL anchor. Mode-seeking—eyes on the fattest fish. Cost: reward scale flows directly into gradients.
• DPO (Rafailov et al., 2023): winner–loser pairs against an EMA reference. Only reward *ordering* matters; monotone rescaling doesn't move it—naturally robust but insensitive to magnitude.
• GFlowNet (Bengio et al., 2023; Malkin et al., 2022): trajectory balance with terminal reward R(τ) = exp(clip(Σr_k, −10, 10)). Mode-covering—samples the space proportionally to reward, ideal for discovering diverse violations—but demands careful reward calibration.

The Numbers: Thirty-Five to Sixty

Main results, 50 matched scenarios, training-time verifier suite:

| Player | Coverage | yield/100 | B4/S5/B6 share | Multi-policy rate | |---|---|---|---|---| | Random scenarios | 5% | 28 | 100% | 4% | | Zero-shot Qwen3-30B | 35% | 620 | 88% | 64% | | PROBE-GRPO | 60% | 1176 | 77% | 94% | | PROBE-DPO | 60% | 836 | 77% | 88% | | PROBE-GFlowNet | 45% | 1016 | 83% | 78% |

Three readings:

• A small model fishes out big flaws 🎣: random probing found 1 of 20 policies; zero-shot 30B found 7; the trained 4B auditor found 12—coverage lifted from 35% to 60%. Craft beats physique.
• 94% multi-policy rate: the GRPO auditor triggers simultaneous multi-policy violations in 94% of evaluation episodes—one fish, many bites.
• Share drops, entropy rises: the easy-policy share falls from 88% to 77%; violation-distribution entropy climbs from 2.45 to above 3.1—training taught it to abandon shortcuts.

Output ≠ Coverage

The most striking ablation: GRPO's dedup-only variant pushes yield to 1914/100 with 100% multi-policy rate—yet coverage stays at 55% 😬. Ticketing by the thousands, but always the same eleven streets. High yield is not high discovery—hence the headline metric is suite-level coverage, with yield read alongside concentration and entropy.

Change the Reward, Change the Temperament

Calibrated rewards lift training-time yield 63% for GRPO and 78% for GFlowNet; DPO barely moves. DPO eats only rankings; GRPO and GFlowNet gradients carry raw reward values—a 63–78% swing says reward design isn't hyperparameter tuning; it's drafting business regulation. GFlowNet is subtler: replace per-policy credit with generic diversity bonuses and coverage drops from 45% to 35%. Reassurance: with deterministic scoring and per-policy credit intact, all reward configurations shift per-optimizer coverage by at most one policy; halving or doubling γ_r moves GRPO coverage by at most one policy.

The Real Ceiling: Counter Depth

Where do the missing 40% hide? Tool depth. The target enters search states hundreds of times (461/341/420) but book only 3/1/3 times, hold 6/4/1, cancel 0/0/0—a conservative clerk, meaning deeper operational safeguards were never exercised. Difficulty tiers: easiest—turn-level omissions B4/S5/B6; medium—B2, P1, S4, S6, P3, P4; hardest—B3, B5, O1–O4, each requiring the target to enter specific operational states.

Qualitatively, the trained auditor has two signature moves: tempting under-specification—plausible but incomplete requests that force premature action (e.g., asking for flights to LLW without naming an origin; the target verbally asks for the origin yet calls search_flights with a missing required field—a schema failure invisible in the public response) 🪤; and policy stacking—combining budget, date flexibility, passenger count, urgency, and modification intent so multiple policies trip together. Failure mode: shallow persistence—circling nearby cities and farming repeat B4/S5/B6 when the target keeps reporting no flights.

Don't Trust the Judge; Re-Score It

Human calibration of the 4 semantic judges found a firing-rate asymmetry between false positives and false negatives—and judges won't admit it. So the team made a laudable call: after calibration, all methods are re-scored with the 16 deterministic predicates; judges remain only as training-time coaches. Under deterministic re-scoring, Probe covers 55% vs 5% for the static suite (an admittedly harsh comparison—static scripts weren't written for these 50 scenarios—but it proves random walk-ins and trained craft are different species).

Stronger baselines (static few-shot, cooperative, diversity-seeking) all lose to Probe. Crucially, transfer works: to an unseen target family (Gemma) and to a new domain (τ-bench, Yao et al., 2024). The learned thing is a craft, not muscle memory for one shop. Boundary: the auditor is a detector, not a judge—findings still need human adjudication.

Closing a Trilogy: Cultivate an Evolving Inspectorate

Post one diagnosed: Shortcut Sensitivity—the guard-evaluation ruler itself is crooked. Post two repaired: RADAR—after confidence comes acceptance and audit; fixed ≠ trusted. This post institutionalizes: rather than repairing a ruler that may always go crooked again, maintain a standing, evolving inspectorate—active, periodic, conversational probes that fish violations to the surface.

The paper's last word is co-evolution: auditors expose failures, agents improve from feedback, auditors evolve against stronger agents. Red-team/defense co-evolution is old news in text safety; PROBE moves it onto procedural tool use—a world with a standing inspectorate is far more reliable than one without.

The trilogy: diagnosis—the ruler goes crooked; repair—fixed ≠ trusted; institutionalization—cultivate an evolving inspectorate. The morning question now has an answer: who keeps account of that perfect Bangkok trip? A 4B undercover inspector, sitting at the counter, smiling: "Book me a Friday flight to Bangkok, nothing too pricey."

Feynman said: a rule we haven't sent someone to try to break doesn't count. This paper turned that sentence into an algorithm.

References

1. Mishra, K., Sharma, A., Lukas, N., & Lahlou, S. (2026). PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents. arXiv:2610.22124. 2. Yao, S., et al. (2024). τ-bench. arXiv:2406.12045. 3. Shao, Z., et al. (2024). DeepSeekMath. arXiv:2402.03300. (GRPO) 4. Rafailov, R., et al. (2023). Direct preference optimization. NeurIPS 2023. 5. Bengio, Y., et al. (2023). GFlowNet foundations. / Malkin, N., et al. (2022). Trajectory balance. ICML 2022. 6. Arkin, B., et al. (2005). Software penetration testing. 7. Perez, F., et al. (2022). Red teaming language models with language models. EMNLP 2022. 8. Andriushchenko, M., et al. (2025). AgentHarm. ICLR 2025. 9. Dettmers, T., et al. (2023). QLoRA. NeurIPS 2023. 10. Yang, A., et al. (2025). Qwen3 technical report.

arXiv:2610.22124 | papers.cool #25