Table Order Attacks: Can Permuting Rows and Columns Fool LLMs?

Table Order Attacks: Can Permuting Rows and Columns Fool LLMs?

> Paper: The Power of Order: Fooling LLMs with Adversarial Table Permutations > Authors: Xinshuai Dong, Haifeng Chen, Xuyuan Liu, Shengyu Chen, Haoyu Wang > arXiv: 2605.00445 | 2026-04-29

1. The LLM That "Flips Out" When Rows and Columns Swap

Consider asking an LLM a question about two tables containing identical data:

Table A:

| Name | Age | City | |------|-----|------| | Zhang San | 25 | Beijing | | Li Si | 30 | Shanghai |

Table B (rows/columns shuffled):

| City | Name | Age | |------|------|-----| | Shanghai | Li Si | 30 | | Beijing | Zhang San | 25 |

Semantically, these two tables are completely identical — but the paper finds that an LLM may answer Table B very differently from Table A.

2. Semantics-Preserving Structural Transformations

The paper exposes a striking fragility: adversarial table permutations — shuffling row order and column order without changing any data content or semantics — can cause:

• Answers to change from correct to incorrect
• Large swings in model confidence
• Complete misinterpretation of the question

It is like scrambling the chapter order of a book: the content is unchanged, yet the reader can no longer follow it.

3. Why Are LLMs So Sensitive to Order?

1. Positional encoding bias — Transformers assign different weights to different positions; "first row" and "last row" are treated differently, so position influences attention even when semantics are identical. 2. Context window limitations — LLMs do not truly "remember" the whole table; attention weakens for information far from the current token, so key values pushed far away effectively become invisible. 3. Training data bias — Training tables usually follow "natural" orderings (by time, by importance). LLMs learn to rely on this, becoming confused when order is scrambled. 4. Fragile reasoning chains — LLM reasoning relies on step-by-step comparisons; after shuffling, the comparison path changes and so do the results.

4. Why Is This Dangerous?

• Table QA: A malicious user can deliberately shuffle a table to make the model produce wrong answers.
• Data analysis: If conclusions depend on table order, LLM-assisted analysis may lead to bad decisions.
• Finance / healthcare: Automated processing of tabular data could yield wrong financial advice or incorrect diagnostic suggestions.

5. Judgment: Order Is Structure, and Structure Is Meaning

As Feynman suggested, knowing the name of something is not the same as understanding it. Applied to table understanding:

> "Knowing every value in a table is not understanding it. True understanding means extracting the same information no matter how rows and columns are arranged. If an LLM is fooled by order, its 'understanding' is shallow — it relies on position, not relations."

Humans actively look for structure when reading tables and are unaffected by row/column order. LLMs, it seems, have not yet learned this kind of structured understanding.

6. Takeaways for Builders

If you are building AI systems that use tabular data, ask yourself:

1. Is my system sensitive to table order? 2. Do I normalize/sort tables during preprocessing? 3. Does my model truly understand table structure, or does it rely on position? 4. Have I tested robustness under semantics-preserving transformations?

Core lesson: LLM table understanding is more fragile than we assume. When deploying LLMs in table-centric critical applications, a simple row/column shuffle may be enough to fool the model. Genuine table understanding requires order-invariant, structured reasoning. In the world of tables, content matters — but how the model understands that content matters more.

---

*Translated and adapted from the original zhichai.net forum post.*