Adversarial Table Permutations: How Row and Column Reordering Can Fool LLMs

> Paper: The Power of Order: Fooling LLMs with Adversarial Table Permutations > Authors: Xinshuai Dong, Haifeng Chen, Xuyuan Liu, Shengyu Chen, Haoyu Wang > arXiv: 2605.00445 | 2026-04-29

1. The LLM That Changes Its Answer When You Shuffle Rows and Columns

Imagine asking an LLM a question about a table:

Table A:

| Name | Age | City | |------|-----|------| | Zhang San | 25 | Beijing | | Li Si | 30 | Shanghai |

Table B (rows and columns shuffled):

| City | Name | Age | |------|------|-----| | Shanghai | Li Si | 30 | | Beijing | Zhang San | 25 |

To an LLM, these two tables are semantically identical.

But the paper finds: the LLM's answers to Table B may be completely different!

2. Semantics-Preserving Structural Transformations

The paper reveals a striking fragility:

Adversarial table permutations:

• Shuffle row order
• Shuffle column order
• Change no data content
• Change no semantics

Yet LLM performance may:

• Go from correct to incorrect answers
• Show large swings in confidence
• Even completely misread the question

It's like shuffling the chapters of a book and finding the reader can no longer understand it—even though the content is exactly the same.

3. Why Are LLMs So Sensitive to Order?

Analysis of the causes:

1. Positional encoding bias

• Transformer positional encodings assign different weights to different positions
• The "first row" and "last row" of a table are treated differently
• Even with identical semantics, position influences attention

2. Context window limitations

• LLMs don't truly "remember" an entire table
• Attention weakens for information far from the current token
• Once key information is "pushed" far away, the model effectively can't see it

3. Training data bias

• Tables in training data usually have a "natural order" (by time, by importance, etc.)
• LLMs learn to rely on such ordering
• When the order is shuffled, the model gets confused

4. Fragile reasoning chains

• LLM reasoning depends on step-by-step comparisons
• Shuffling changes the comparison path
• And the reasoning outcome changes with it

4. Why Is This Dangerous?

Real-world scenarios:

Table QA:

• Users upload tables and ask questions
• A malicious user could deliberately shuffle rows/columns
• To make the model produce wrong answers

Data analysis:

• LLM-assisted analysts
• If table order influences conclusions
• Decisions may rest on faulty analysis

Finance / healthcare:

• Automated processing of tabular data
• Order attacks could cause:
• Incorrect financial advice
• Incorrect diagnostic suggestions

5. A Feynman-Style Judgment: Order Is Structure, and Structure Is Meaning

Feynman famously noted that knowing the name of something is not the same as understanding it.

Applied to table understanding:

> Knowing every value in a table is not the same as understanding the table. True understanding means extracting the same information no matter how rows and columns are arranged. If an LLM can be fooled by order, its "understanding" is shallow—it relies on position, not relationships.

This also reminds us:

• Humans actively look for structure when reading tables
• They are not affected by row/column order
• But LLMs have apparently not yet learned this kind of "structured understanding"

6. Takeaways

If you're building AI systems that use tabular data, ask yourself:

1. "Is my system sensitive to table order?" 2. "Do I normalize/sort tables in preprocessing?" 3. "Does my model truly understand table structure, or does it just rely on position?" 4. "Have I tested robustness under semantics-preserving transformations?"

The paper's core lesson: LLM table understanding is more fragile than we assumed.

When deploying LLMs in critical table-based applications, be aware that a simple row/column shuffle can fool them. Genuine table understanding requires order-invariant, structured reasoning.

In the world of tables, content matters—but how the content is understood matters more.