Judged in Company: The Same Rating Changes by 11–22% When You Shuffle the Batch

A Counterintuitive Finding

You ask a large language model to judge 25 social media comments for harmfulness. You pack all 25 into a single prompt — the industry-standard approach that saves money and tokens.

Then you run a seemingly harmless experiment: re-randomize the grouping of the same 25 comments into new batches of 25, and ask the model again.

Result: 11% to 22% of the verdicts flip.

The model didn't get dumber, the prompt didn't change, the comments didn't change. Only the company changed — who sits next to whom.

Stranger still: when you compare AUROC (the standard classification metric) between individual scoring and batched scoring, the difference is only -0.033 to +0.010. In other words, the model looks just as accurate overall — but a fifth of individual verdicts get overturned.

This is the core finding of the paper "Judged in Company": LLM judgments are not separable — the same item receives different verdicts depending on its companions.

The paper is by Peng Xie and Amr Alanwar of the Technical University of Munich, published October 29, 2026.

Experimental Design: Randomness as a Tool

The researchers did three simple but brutal things.

Tasks: three judgment tasks — harmfulness classification of social media comments, short-answer grading, and review star-rating — covering classification, scoring, and ranking, the three typical LLM-as-judge scenarios.

Models: five open-source models with 4–9B parameters (Llama, Qwen, Mistral families), plus four 14–32B models — covering mainstream sizes.

Core operation: assign a fixed set of 2000 items to different random batches of 25 each, then check whether the same item receives consistent verdicts across batches.

An analogy: a teacher grades 25 exam papers. You shuffle the same 25 papers into different groups of 25 and have the teacher grade again — and the same paper gets a different score depending on which papers accompany it.

What Does 11–22% Mean?

To calibrate the magnitude, the authors ran control conditions:

• Renaming item keys (e.g., comment_1 → item_A): verdict changes of 4.8–8.7%
• Rephrasing instructions: 3.5–9.5%
• Re-randomizing batches: 11–22%

So batch membership matters by 6–13 percentage points more than question phrasing and field naming combined.

For larger 14–32B models, the figure is 6–15% — somewhat better, but still substantial.

Why Does AUROC Barely Move?

This is the paper's most elegant point. AUROC measures overall ranking ability — whether the model places harmful items above harmless ones — and it is nearly identical between batched and individual scoring.

But AUROC is an aggregate metric. It tells you the model separates well on average; it says nothing about whether a specific item's verdict is stable.

Analogy: a weather forecaster predicts tomorrow's high temperature with 90% accuracy — yet the binary "will you need an umbrella tomorrow" decision flips 20% of the time just by reordering the same data. Temperature accuracy masks the instability of the umbrella decision.

The paper offers a mathematical account: the flip rate decomposes exactly into order, slot, and membership components.

• Order: the arrangement of items within a batch
• Slot: the item's position in the batch
• Membership: which items are grouped together

Of these, membership dominates, contributing a flip rate of 0.11–0.22 — and it cannot be voted away: even majority voting over 9 permutations preserves 52–60% of the flips.

Two Channels: Neighbor Text and Prior Verdicts

The researchers further disentangle why neighbors affect judgments, identifying two channels:

Channel 1: neighbor text itself. Place a mild comment next to 24 extremely violent ones, and the model's harmfulness score for the mild comment rises — a contrast effect, long known in human psychology, now observed in LLMs.

Channel 2: the model's own prior verdicts. Once the model has labeled the first 10 comments as harmful within the prompt, its verdict on the 11th is anchored by its earlier judgments — contextual inertia.

These two channels compound, producing different verdicts for the same item across batches.

Why It Matters

1. The separability assumption behind evaluation is wrong.

Nearly all LLM evaluation pipelines assume that a given item's verdict is independent and reproducible across prompts. This paper shows that assumption fails: change the batching, and 20% of verdicts flip.

2. Aggregate metrics hide decision instability.

AUROC differs by 0.03 — negligible. But whether a specific user's comment gets flagged may hinge entirely on which comments happen to share its batch. Direct consequences for content moderation, educational scoring, and review analysis.

3. Voting doesn't fully solve it.

Majority voting across permutations reduces order and slot effects but cannot eliminate membership effects. Voting is a mitigation, not a solution.

My Take: Another Instance of the Evaluation Blind-Spot Law

This paper fits a recurring pattern: aggregate metrics systematically mask local instability.

• Standard benchmark accuracy masks under-encoded spatial relations in VLM representations (CASP-VL/EICS)
• Perplexity masks the collapse of critical circuits after distillation (Distillation)
• AUROC masks a 20% verdict flip rate in batched scoring (this paper)

The right question isn't "is the model accurate?" but "accurate in which contexts, unstable in which contexts?"

The mathematical decomposition is especially clean: flip rate = f(order, slot, membership), with membership irreducible by voting. This isn't a matter of models not being good enough — it's a structural flaw in the batched-evaluation paradigm itself.

A practical takeaway: if in your product a user's comment gets deleted depending on which comments it happens to be batched with, your moderation system carries ~20% randomness. That's not the model's fault — it's how you're using it.

---

Paper: arXiv:2610.40241 Code: github.com/PX-CPS/Judged_in_company