The Embarrassing Fact: The Strongest Models Can't Copy Strings Correctly
In July 2026, a paper from Tsinghua and Peking University teams delivered a hard-to-believe conclusion: GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro—frontier models that can solve competition math, write code, and perform complex reasoning—frequently fail at a seemingly trivial task: copying a string from context verbatim.
Not strings too long for the context window, not special encodings. Just ordinary binary strings and Python lists. The model only needs to do one thing: see the character at position \(i\), and emit it unchanged at output position \(i\).
Yet accuracy often falls below 50%. The longer the input, the worse the errors.
It's like a university student who can solve calculus but misses half the characters when copying a line of text. Where does the problem lie?
The Problem Isn't "Capability"—It's Positional Encoding
The researchers ran a clever diagnostic. Their hypothesis: when LLMs copy strings, they may not locate the \(i\)-th input character via "absolute position \(i\)" at all, but instead take a shortcut—matching local context.
What does that mean? Imagine the string to copy is ...A B A B A B.... Seeing A B before the current position, the model thinks: "Where have I seen A B before? Oh, just up ahead, and it's followed by A—so I'll copy A." This is the well-studied induction head mechanism in Transformers.
This mechanism works well on most tasks, but it has a fatal weakness: when local contexts repeat, the model confuses which occurrence to copy from. The longer the string, the more repeated patterns, the more errors.
The root cause is the Transformer's 1D positional encoding (RoPE): all tokens are lined up in a single row with one-dimensional position information. The model has no concept of "rows" and "columns" and can only localize via local context—which repeats in long strings.
2D-RoPE: Viewing Text as a Grid
The researchers' fix is strikingly simple: reorganize the 1D sequence into a 2D grid.
Specifically, text is no longer a single line but folded into a matrix—say, 64 tokens per row, wrapping when overflowing. Each token thus gets two position IDs: a row ID and a column ID, and RoPE's rotation angles are determined jointly by both.
In this 2D view, copying becomes trivially easy: output positions and input positions sit in the same row, with a fixed column offset. The model only needs to learn "take the token \(k\) columns back in the same row," enabling exact copying regardless of input length.
It's like moving from "fuzzy matching in a 1D river" to "precise lookup by column index in an Excel sheet."
Results: 100x Length Extrapolation, Still 100%
Synthetic results are stunning:
- Trained only on strings of length 128
- Tested on inputs up to 12,800 (100x longer)
- Standard 1D-RoPE shallow transformers fall to near 0% accuracy
- 2D-RoPE transformers maintain near-100% copy accuracy
Crucially, large-scale pretraining confirms it. Training language models from scratch on the DCLM dataset, up to 1.4B parameters, comparing standard RoPE vs 2D-RoPE:
| Model | Params | Train tokens | Short-input copy | Long-input copy | |-------|--------|--------------|------------------|-----------------| | RoPE | 730M | 15B | 97.0% | 7.8% | | 2D-RoPE | 730M | 15B | 99.4% | 76.7% | | RoPE | 1.4B | 28B | 99.6% | 8.7% | | 2D-RoPE | 1.4B | 28B | 100.0% | 98.1% |
On a 730M model trained for 100B tokens, 2D-RoPE reached 100% copy accuracy on the longest inputs, while standard RoPE managed only 1.8%.
Just as important, 2D-RoPE shows no regression on standard benchmarks like commonsense reasoning (CSR)—it doesn't trade reasoning ability for copying ability; it does both well.
Why This Matters
The story is fascinating not because it's "yet another positional encoding variant," but because it exposes a structural blind spot:
Many Transformer "capability deficits" may stem not from insufficient parameters, data, or training, but from the inductive bias of positional encoding steering the model toward shortcuts from the very start.
The model didn't "fail to learn to copy"—the positional encoding handed it an easier path (local context matching) that works on short inputs but collapses on long ones. Change the positional encoding, and the same model immediately gets it.
This suggests a deeper analogy: sometimes it's not that people aren't smart enough, but that the shape of the tool constrains how one thinks. Viewing text in 1D, copying is "find a match"; viewing text in 2D, copying is "look up by coordinates." Same information, different organization, vastly different difficulty.
The 2D-RoPE authors themselves say they hope this work encourages more exploration of multi-dimensional positional encodings. After all, humans don't read text as a single line either—we have paragraphs, rows, columns, layouts. Perhaps models should too.
---
Paper: Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D