Black-Box Detective: A Four-Stage Protocol for Auditing Anonymous AI Models
> *"When models hide their names, whom can we still trust?"*
This post is an English translation of a Chinese forum discussion of the paper "Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification" (Xi, Y., 2026, arXiv:2608.31142).
Introduction: A Game Without Names
The post opens with a scenario increasingly common in 2025–2026: an anonymous model, codenamed "Aurora-7," appears on a developer platform with no logo, whitepaper, or team information. Such stealth releases create real problems — model identity determines data-processing terms, supply-chain risk assessment, and capability baselines. Yet no validated method existed to identify what is inside the black box, and a model's self-declaration about its own identity is untrustworthy by design.
The Four-Stage Forensic Protocol
Stage 0: Temporal Archaeology
Reconstruct the model's launch-time configuration from Internet Archive snapshots:
1. Determine the model's first appearance date (a timestamp for the "masked knight") 2. Compare configuration drift between preview and production versions — models may behave more freely in preview and more conservatively in production 3. Trace version iteration history
*Analogy:* A vintage camera collector verifies a supposed "1950s German Leica" by checking the shop's archived web history — discovering it was only listed in 2023 and was originally labeled a "replica." Time doesn't lie, but current pages might.
Stage 1: Configuration Fingerprinting
Like a sommelier identifying an unlabeled wine, auditors measure distinctive traits and match them against known models in platform catalogs:
- Context window size (4K, 32K, 128K tokens?)
- Output ceiling per response
- Reasoning style (direct answer vs. explicit chain-of-thought)
- Modality support (text, image, audio, video)
- Math reasoning problems — each model has a distinct solution style and characteristic error patterns
- Creative writing tasks — narrative style, vocabulary choice, plot direction
- Ethical dilemma scenarios — different training backgrounds produce different instinctive responses
- 7 exact matches
- 2 precision-differences (correct family, minor version discrepancy)
- 1 partial match
- 0 counter-directional errors — the protocol never misidentified one model as another
- Data privacy — without knowing who built the model, users can't know how data is handled
- Supply-chain security — enterprises need to assess vendor stability and support commitments
- Capability expectations — different models excel at different tasks
- Xi, Y. (2026). *Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification*. arXiv:2608.31142.
Stage 2: Tokenizer "DNA Testing"
The protocol's most ingenious component. Each model's tokenizer is a unique fingerprint. The cross-length differential technique:
1. Send a short prompt, observe tokenization-related behavior 2. Send a long prompt, observe again 3. Compare the differential — different tokenizers show measurable divergence on long vs. short inputs
This rejects short-prompt collisions: models may appear similar on trivial questions, but their tokenizers betray them on long, complex text. Like an accent that only emerges in extended conversation, the model's "language DNA" surfaces when utterances grow long.
Stage 3: Behavioral Profiling
Like identifying a painter's brushstrokes, behavioral probes examine *how* a model thinks rather than what it says:
Experimental Results
Tested on 10 known models:
In identity verification, avoiding false attributions matters more than complete coverage.
Flagship Case: The GLM-5.3 Prediction
On August 23, 2026, a codenamed model appeared with no identity information. Running the four-stage protocol, the team concluded it belonged to the GLM-5.3 version line. Official disclosure later confirmed the family and version-line inference was fully correct, and the Flash-variant hypothesis was consistent post-reveal (though no deployment variant was pre-asserted).
Why This Matters
The post also draws parallels to forensic science: tokenizer analysis ≈ DNA analysis, configuration fingerprinting ≈ fingerprints, behavioral probes ≈ criminal profiling, archive reconstruction ≈ timeline building. Multiple independent evidence chains pointing to the same conclusion exponentially raise confidence — no single "silver bullet."
Acknowledged Limitations
The author is candid about boundaries:
1. Not end-to-end blind identification — tests verified declaration consistency; fully blind identification needs further research 2. Deployment-variant uncertainty — family and version line are identifiable, but specific variants (Flash/Pro/Nano) may not be distinguishable 3. Adversarial evasion — a party intent on concealment could modify tokenizers or tune behavior to escape detection