Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ Ordering Effect Explained by the LR Schedule

Forum topic · 小凯 · 2026-10-05

Summary

A new arXiv paper (2610.00234) by Wenhui Chen studies what data ordering writes into model parameters when training on the same problems expressed under two incompatible but individually correct conventions. The paper proves that the learning-rate schedule is the averaging operator that decides the answer: arrangement and schedule enter the ordering effect as separate multiplied factors, with arrangement contributing only a block period and the schedule only how much weight the endpoint places on any moment of the run. Decaying schedules cannot combine a large step size with an uncontracted remainder, while constant schedules do so at the last step. Experiments with ten permutations under identical budgets show that under constant learning rate, internal allocation differences span 0.2221 and vary monotonically with chunking, whereas cosine schedules collapse ten runs into only two distinguishable states. Accuracy summed over both conventions stays constant within 9.7% while allocation shares range from 0.04 to 0.87, so the measured 12.29σ permutation switch is exactly zero under convention-agnostic metrics. Prompt-based convention labeling collapses the switch and reaches 87.5% of the union upper bound.

Paper Overview

Field: Machine Learning Author: Wenhui Chen Published: 2026-10-05 arXiv: 2610.00234

Abstract

> Train a model on the same problems written under two incompatible conventions, both correct, and ask what the ordering of that data writes into the parameters. The learning-rate schedule is not a background condition for that question. It is the averaging operator, and it decides the answer. We prove a bound in which the arrangement and the schedule enter the ordering effect as separate multiplied factors: the arrangement only as a block period, the schedule only as how much weight the endpoint can place on any one moment of the run. A decaying schedule cannot put a large step size and an uncontracted remainder at the same moment; a constant one does exactly that at the last step. That decay moderates ordering effects has been reported in pretraining; the mechanism, the separation, an...

Key Contributions (from the forum summary)

The learning-rate schedule acts as an averaging operator determining what data ordering writes into parameters, not merely a background condition.
A proven bound: arrangement and schedule multiply independently — arrangement enters only via block period; schedule enters only via endpoint weighting of the run.
Decaying schedules cannot combine a large step size with an uncontracted remainder at the same moment; constant schedules achieve exactly this at the last step.
Controlled experiments: same corpus, ten permutations, same budget, two families differing only in lr_scheduler_type, two runs each.
Under a constant rate, internal allocation differences span 0.2221 with a lower bound of 11.63 across comparisons, monotonic in chunking degree.
Under a single cosine schedule (used in all published results), the same ten-arm setup occupies only two distinguishable states, though its own resolution could distinguish about ten.
"Order matters" and "order doesn't matter" are two ends of the same knob (the schedule).
Across twelve arms, acc_A + acc_B stays constant within 9.7%, while allocation shares range from 0.04 to 0.87 — the measured 12.29σ permutation switch is exactly zero under convention-agnostic metrics.
Labeling the convention in the prompt collapses the switching and reaches 87.5% of the union upper bound.

Original Link

https://arxiv.org/abs/2610.00234

--- *Auto-collected on 2026-10-05*