The Embarrassing Fact: The Strongest Models Cannot Copy Strings Correctly
In July 2026, a paper from Tsinghua and Peking University teams delivered a hard-to-believe conclusion: GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro—models that can solve Olympiad math, write code, and perform complex reasoning—frequently fail at a seemingly trivial task: copying a string from the context verbatim.
Not strings too long for the context window, nor specially encoded inputs. Just ordinary binary strings and ordinary Python lists. The model only needs to do one thing: see the character at position \(i\) and output it at position \(i\).
Yet accuracy often drops below 50%. The longer the input, the worse the errors.
It is like a college student who can solve calculus but drops half the characters when copying a line of text. Where does the problem lie?
The Problem Is Not "Capability" but Positional Encoding
The researchers ran an elegant diagnostic. They hypothesized that when LLMs copy strings, they do not locate the input's \(i\)-th character via "absolute position \(i\)" at all; instead they take a shortcut—matching local context.
What does that mean? Imagine the string to copy is ...A B A B A B.... Seeing A B before the current position, the model thinks: "Where have I seen A B before? Oh, just up ahead, and it is followed by A, so I'll copy A." This is the well-studied induction head mechanism in Transformers.
This mechanism works well on most tasks, but it has a fatal weakness: when local contexts repeat, the model confuses which occurrence to copy from. The longer the string, the more repeated patterns, and the more position errors accumulate.
The root cause is the Transformer's 1D positional encoding (RoPE): all tokens line up in a single queue, and position information is one-dimensional. The model has no concept of "rows" and "columns," so it can only localize via local context—and local context repeats in long strings.
2D-RoPE: Viewing Text as a Grid
The researchers' solution is remarkably simple: reorganize the 1D sequence into a 2D grid.
Concretely, text is no longer laid out in one line but folded into a matrix—say, 64 tokens per row, with the remainder wrapped to a new line. Each token thus gets two position IDs: a row ID and a column ID. RoPE's rotation angles are determined jointly by both.
Under this 2D view, copying becomes trivially simple: the output position and input position share the same row, with a fixed column offset. The model only needs to learn "take the token \(k\) columns back in this row," and it can copy precisely regardless of input length.
It is like moving from "fuzzy matching in a 1D river" to "indexing by column number in an Excel spreadsheet." The former is fuzzy matching; the latter is exact indexing.
Experimental Results: 100x Length Extrapolation at ~100% Accuracy
The synthetic results are striking:
- Models trained only on strings of length 128
- Tested on inputs up to 12,800 tokens (100x longer)
- Standard 1D-RoPE shallow Transformers fell to near 0% accuracy
- 2D-RoPE Transformers maintained near 100% copy accuracy
More importantly, the authors validated this at pretraining scale. They pretrained language models from scratch on the DCLM dataset up to 1.4B parameters, comparing standard RoPE and 2D-RoPE:
| Model | Params | Training tokens | Short-input copying | Long-input copying | |------|------|-----------|-----------|-----------| | RoPE | 730M | 15B | 97.0% | 7.8% | | 2D-RoPE | 730M | 15B | 99.4% | 76.7% | | RoPE | 1.4B | 28B | 99.6% | 8.7% | | 2D-RoPE | 1.4B | 28B | 100.0% | 98.1% |
On a 730M model trained on 100B tokens, 2D-RoPE reached 100% copy accuracy on the longest inputs, while standard RoPE managed only 1.8%.
Crucially, 2D-RoPE showed no regression on standard benchmarks like common-sense reasoning (CSR)—it does not trade reasoning ability for copying ability; it improves both.
Why This Matters
The story is compelling not because it is "yet another positional encoding variant," but because it exposes a structural blind spot:
Many of the Transformer's "capability deficits" may stem not from insufficient parameters, data, or training, but from the inductive bias of the positional encoding steering the model toward shortcuts from the very start.
The model did not "fail to learn copying"—the positional encoding offered it an easier path, local-context matching, which works on short inputs but collapses on long ones. Swap the positional encoding, and the same model immediately learns it.
This suggests a deeper analogy: sometimes it is not that people are not smart enough, but that the shape of the tool constrains how they think. Viewing text in 1D makes copying a "matching" problem; viewing it in 2D makes it "coordinate lookup." Same information, different organization, vastly different difficulty.
The 2D-RoPE authors themselves say they hope the work encourages more exploration of multi-dimensional positional encodings. After all, humans do not read text as a single line either—we have paragraphs, rows and columns, layouts. Perhaps models should too.
---
Paper: Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D