Repeated-Token Fingerprinting: A Surprising Model Fingerprinting Method from a Model's Own Tokens

Forum topic · ✨步子哥 · 2026-09-28

Summary

An independent researcher's paper, 'Where a Model Sends Its Own Repeated Token' (Michael Paulun, 2026), proposes a black-box model fingerprinting technique: feed a model its own tokens repeated many times and observe its outputs. Most tokens self-continue, but non-self-continuing tokens are routed to specific destinations, forming a 'destination map' unique to each model. The first registered hypothesis—using fixed-point token sets with Hamming distance as fingerprints—failed due to a cardinality trap (correlation 0.9128 between Hamming distance and set sizes). The destination-map approach succeeded: across 19 models and 7 tokenizers, family attribution reached 0.8333 accuracy (6x baseline), and architecture-class attribution achieved 0.90–1.0 balanced accuracy after honest corrections. The fingerprint proved robust to 8-bit quantization (agreement 0.9004) but collapsed at 4-bit per-channel quantization (0.0098). Analogized to DNA barcoding, the method captures model-internal computational identity without weight access.

Repeated-Token Fingerprinting: Where a Model Sends Its Own Repeated Token

A Seemingly Boring Experiment

What happens if you feed a language model a single token repeated many times—"apple apple apple apple..."?

Intuition says the model should just output "apple" again. But in September 2026, independent researcher Michael Paulun published a paper doing exactly this experiment and found something surprising: most tokens do "self-continue," but a substantial fraction do not. For those non-self-continuing tokens, the model routes them to a specific next token—and this routing pattern is unique to each model, like a fingerprint.

The paper, *Where a Model Sends Its Own Repeated Token* (arXiv:2609.31181), does something conceptually simple but operationally meticulous: fingerprinting models using their own token behavior.

A Story in Two Halves

Half one: fixed points. Give the model a token repeated several times; if its predicted next token is the same token, that token is a "fixed point." Across 12 Pile-trained models, the intersection of shared fixed points was 3,471 strings.

Half two: the destination map. For tokens that do *not* self-continue, where does the model send them? This forms a source-to-destination mapping.

The registered hypothesis was that fixed-point sets could serve as fingerprints, compared via Hamming distance. This hypothesis failed, and instructively so.

Why It Failed: The Cardinality Trap

| Model pair | Hamming distance | |---|---| | pythia-410m vs -deduped (corpus dedup) | 2 | | Same weights, different precision | 0 | | pythia-410m vs gpt-neo-125m | 276 | | pythia-410m vs mamba-370m | 661 | | pythia-410m vs rwkv-4-430m | 440 |

Superficially this distinguishes families. But there was a fatal flaw: the correlation between Hamming distance and the sum of the two set sizes was 0.9128—about 83% of the "difference" merely reflected set size, not identity. For sparse sets, Hamming distance |A| + |B| − 2|A∩B| is dominated by set sizes.

Worse, pythia-410m vs its deduped variant showed zero difference under strict thresholds—the fingerprint could not distinguish "training-corpus difference" from "numerical-precision noise."

Half Two Wins: The Destination Map

Comparing how often two models send non-self-continuing tokens to the same target (agreement rate):

| Model pair | Agreement rate | vs. baseline | |---|---|---| | Same weights, different precision | 0.7127 | — | | pythia-410m vs -deduped | 0.6355 | 1.27× | | pythia-410m vs gpt-neo-125m | 0.3902 | 2.12× | | pythia-410m vs mamba-370m | 0.3493 | 2.26× | | pythia-410m vs rwkv-4-430m | 0.3393 | 2.30× |

Key result: the correlation between agreement rate and set sizes dropped from 0.9128 to −0.0932—the cardinality trap eliminated by design, not statistical adjustment. Agreement wasn't driven by token frequency: shared destinations reflect similar internal mechanisms.

Scaling to 19 Models

The cohort grew to 19 models spanning 7 tokenizers. Three pre-registered questions:

1. Family attribution (rank-1 leave-one-out): 0.8333 accuracy vs 0.1389 baseline—6× random. 2. Architecture-class attribution (Transformer vs RNN vs hybrid): balanced accuracy 1.0 vs 0.7895 baseline. After excluding each model's single most common destination, this honestly drops to 0.90. 3. Family vs tokenizer: family lift 0.2031 vs tokenizer lift 0.1205—family > tokenizer, meaning the fingerprint captures internal computation, not tokenization style.

Quantization Robustness

8-bit quantization: agreement 0.9004 with full precision—*less* impact than training-corpus deduplication (0.6353).
4-bit per-channel quantization: agreement collapses to 0.0098—the fingerprint fails.
4-bit grouped quantization (groups of 128 channels, GPTQ/AWQ-style): 0.1812—an order of magnitude better but still far below 0.6353.
The precision floor is per-model, not global: bfloat16 vs float32 changed 2% of destinations for one model and 80% for another. Within Pythia, precision robustness rises monotonically with scale (70M: 0.201 → 1B: 0.8989), but this fails outside the family—gpt-neo-125m is more robust (0.9569).

Why This Matters

The method is a third category beyond weight fingerprints (requires white-box access) and behavioral fingerprints (sensitive to prompts/sampling): a structural behavioral fingerprint. It examines not what a model *says* but how it *moves through token space*—all via black-box queries only.

It also has a self-referential structure—feeding a model its own tokens and watching how it handles its own products—which formally echoes Gödelian self-reference.

Cross-Domain Analogy: DNA Barcoding

Like DNA barcoding (using a short standard gene segment, e.g., COI, to identify species), this is "model barcoding": a short standard behavior probe that doesn't reveal everything about a model but suffices to classify its family and architecture.

An Open Question: Why Do Some Tokens Self-Continue?

The paper doesn't systematically explore why certain tokens self-continue. A plausible interpretation: self-continuing tokens are those the model is most confident about; non-self-continuing tokens trigger "escape" toward more familiar tokens. If so, the destination map effectively charts each model's uncertainty structure—which is identity-specific, hence the fingerprint.

Honest Limitations

The paper is notably candid:

1. The fingerprint fails under 4-bit quantization—explicitly "scoped to full precision." 2. Mamba and RWKV models confuse each other within the RNN architecture class. 3. A pre-registered hybrid-architecture test case failed to load, leaving an untested boundary for the "perfect separation" claim. 4. The expanded 19-model cohort added no new families—the 7 new models were distractors, so family attribution rests on the original 12.

Closing

The paper's "fail-then-correct" narrative—half one failed, half two succeeded, both pre-registered and honestly reported—is good science. At a time when AI evaluation chases ever-larger benchmarks, this small, elegant experiment deserves to be remembered: a toy-like design that reveals a deep, unexpected structure—every model has its own unique escape pattern when facing uncertain tokens.

---

Paper: *Where a Model Sends Its Own Repeated Token*, arXiv:2609.31181, 2026-09-25 Author: Michael Paulun (independent researcher) Core numbers: 19 models, 7 tokenizers; family attribution 0.8333 (baseline 0.1389); architecture-class balanced accuracy 0.90–1.0 (baseline 0.7895)