FormInv: How Surface Form Flips LLM Answers on Math Benchmarks

Forum topic · 小凯 · 2026-05-29

Summary

A 2026 paper, FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks (arXiv:2605.29001) by independent researcher Nishal Thomas and Noel Thomas (MBZUAI), shows that frontier LLMs systematically flip answers on logically equivalent rewrites of math problems. Auditing MathCheck with cross-model consistency uncovered 3.1% semantically wrong rewrites whose removal reshuffled over half of model rankings at under $10. On a new benchmark of 103 Lean4-verified Mathlib4 theorems rewritten into 8 certified equivalence families, Claude Haiku scored 86% accuracy but only 50% semantic consistency rate (SCR), while DeepSeek V3 hit 96.4%/82%. Rankings reversed across rewrite families—the paper's No-Free-Benchmark corollary implies any target ranking can be engineered via rewrite weights. Automatic rewrites contained 47% semantic errors detectable only by cross-model auditing. The tool FormInvSelector enables rewrite-family-aware model selection.

Same Question, Different Form, Flipped Answer: A Measurement Black Hole in Math Benchmarks

> Ask an AI two questions: > > Q1: "For all real numbers $$x$$ , $\sqrt{x} \geq 0$ holds, right?" Q2: "For all real numbers $$x$$ , $0 \leq \sqrt{x}$ holds, right?" > > These are logically equivalent statements—the inequality direction and subject order are swapped, but the mathematical meaning is identical. Any high school student sees they say the same thing. > > Claude Sonnet answered Q1 correctly and Q2 wrong—a 16.7% error rate. GPT-4o answered both correctly—0% error. > > If a benchmark author decides inequalities are always written as $a \geq b$ , Claude scores perfectly. Written the other way, GPT-4o wins. The same models, the same theorems. The author's offhand choice of notation decides which model is "better."

In 2026, independent researchers Nishal Thomas and Noel Thomas systematized this intuition in the FormInv paper (arXiv:2605.29001) and uncovered something deeper: every benchmark designer is unknowingly choosing who wins.

| Item | Detail | |------|--------| | Paper | FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks | | Authors | Nishal Thomas, Noel Thomas | | Affiliations | Independent Researcher & Mohamed Bin Zayed University of AI | | arXiv | 2605.29001 (May 27, 2026; cs.LG; cs.AI) |

1. Same theorem, two phrasings

The paper audits MathCheck (ICLR 2025), testing 129 rewrite groups with 4 models and automatically flagging 4 semantically wrong rewrites (3.1%)—rephrasings that silently changed the math, e.g., stripping units so "20 km" became "20". Removing those 4 flawed rewrites moved GPT-4o from rank 2 to 4, Claude Haiku from 3 to 2, DeepSeek V3 from 4 to 3. A 3.1% rewrite-quality error reshuffled over half the rankings. Cost of the audit: under $10.

2. Eight rewrite families, one flipped map

FormInv itself is built from 103 Lean4-verified Mathlib4 theorems across 7 domains (number theory, algebra, set theory, real analysis, etc.), each rewritten into up to 8 semantically equivalent forms across three certification tiers:

| Tier | Rewrite families | Certification | |------|-----------------|---------------| | T1 (formally certified) | F4 symbol variants, F6 comparison order, F7 definition expansion, F8 equivalent propositions | CAS verification / Lean4 proofs | | T2 (conditionally valid) | F1 syntactic substitution, F2 quantifier variants, F3 active/passive conversion | Template guarantees + manual sampling | | T3 (heuristic) | F5 connective variants | Full expert review |

Nine models were run on 366–811 rewrite items. Claude Haiku 4.5 scored 86% accuracy but only 50% semantic consistency rate (SCR)—half of the theorems it "knows" flip after a rewrite. DeepSeek V3: 96.4% accuracy, 82% SCR—a 10-point accuracy gap hiding a 32-point SCR gap. Accuracy is a flat photo; SCR is an X-ray.

3. Who wins depends on the rewrite family

F6 (comparison order, $a \geq b$ vs $b \leq a$ ): GPT-4o and o4-mini tie for first at 0% failure; Claude Sonnet ranks 7th with a 16.7% failure rate.
F7 (definition expansion, "prime" → "exactly two positive divisors"): rankings flip—Claude Sonnet rises to 3rd (6% failure), GPT-4o falls to 7th (10% failure).

The paper derives a No-Free-Benchmark corollary: since no model among the 9 Pareto-dominates all others across all 8 families, for any target ranking there exists a set of rewrite-family weights $\lambda$ that realizes it. Benchmark designers implicitly choose winners—with zero awareness of doing so.

4. 47% of automatic rewrites were wrong

GPT-4o auto-generated the F5 connective rewrites ("iff" → "exactly when", "just in case"), then a cross-model consistency audit flagged rewrites as suspect when ≥6/9 models answered the original correctly but the rewrite incorrectly. Of 15 flagged items, 7 (47%) were human-confirmed as semantically wrong:

Biconditional overgeneralization: writing one-way implications as "if and only if", asserting false converses
Passive-voice inversion: "0 divides $$n$$ " rewritten as " $$n$$ divided by 0", introducing undefined expressions
Type-context stripping: losing Lean4 scoping constraints, making domain-specific theorems false in broader type universes

Notably, GPT-4o answered its own flawed rewrites "correctly"—the question author and the test-taker share the same blind spots. Only cross-model consistency (enough different models failing together) catches rewrite errors automatically. Extrapolated to MathCheck, the method achieved 100% recall—every manually confirmed rewrite error was flagged.

5. FormInvSelector: a rewrite-aware model picker

If your workload is definition-expansion reasoning (F7), run forminv selector --families unpack: it recommends o4-mini (2.0% failure) over GPT-4o (5.0% failure, ~2.5× worse). If your users frequently flip inequality directions (F6), Claude's 16.7% failure rate means picking GPT-4o (0%). FormInv doesn't tell you which model is "best"—it tells you which model fits your actual input distribution.

6. Honest blind spots

What's solid: T1 families are formally verifiable (CAS/Lean4), ground truth comes from Mathlib4 theorems (no manual labeling), the cross-model audit achieved 100% recall in two independent validations, and the No-Free-Benchmark corollary has rigorous support (8 families, 9 models, no Pareto dominator → rankings are sensitive to family weights—not random, but weight-dependent).

What's uncertain:

Coverage: 103 undergraduate-level Mathlib4 theorems are far from word-problem reasoning (GSM8K style). A preliminary Track B test on 100 harder ntp-mathlib theorems showed reduced (though not significant, p=0.27) correlation, hinting at difficulty-range dependence.
Family space: 8 rewrite families do not exhaust semantic-equivalence rewrites (cross-lingual, cross-cultural, cross-representation); the Pareto landscape could change with more families.
False positive rate: the ≥6/9 threshold's false positives at larger scale are unassessed.
SCR → downstream performance: no conversion function from SCR to real-world solving performance is established.

7. The invisible variable

A benchmark claiming "model X is better than model Y at math" measures only one variable: accuracy. It never measures—and often never considers—semantic invariance, even though mathematical reasoning *is* extracting the same logical structure from any equivalent phrasing. FormInv uses Lean4-verified theorems as ground truth, 8 rewrite families as a formal-equivalence stress test, and cross-model consistency as an audit tool—moving evaluation from "how many questions did it answer" to "is its understanding independent of expression?"

It doesn't crown a best model. It asks three crueler, more honest questions: Which rewrite family are you asking? What does your input distribution look like? Are you sure your benchmark's rewrites themselves aren't wrong? Most benchmarks never ask the first two. We didn't even know to ask the third—until FormInv showed the answer can be 47%.

Key contributions

1. 32-point SCR gap: models nearly identical in accuracy (10pp apart) diverge by 32pp in semantic consistency 2. No-Free-Benchmark corollary: model rankings systematically reverse across rewrite families; any target ranking maps to some rewrite weights 3. Cross-model consistency as automated rewrite auditing: 100% recall at under $10 4. 47% semantic error rate in automatic rewrites 5. FormInvSelector: rewrite-family-aware model selection tool 6. Introduces measurement-invariance theory into LLM math evaluation

Key limitations

Limited theorem coverage (103 undergraduate-level theorems), 8 families don't exhaust semantic equivalence, unknown false-positive extrapolation of the audit, no SCR-to-downstream mapping, and difficulty-range dependence suggested by Track B.

References:

1. Thomas & Thomas, "FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks", arXiv:2605.29001, 2026. 2. Zhou et al., "MathCheck: A Behavioral Testing Framework for Mathematical Reasoning of Large Language Models", ICLR 2025. 3. Ribeiro et al., "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList", ACL 2020. 4. Northcutt et al., "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks", NeurIPS 2021. 5. Cronbach et al., "The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles", Wiley, 1972.