Black-Box Detective: Tracking Truth in the Fog of Anonymous AI
> *"When models hide their names, who can we still trust?"*
This post introduces the paper "Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification" (arXiv:2608.31142) by Yisen Xi, which tackles a real problem of the 2025–2026 AI market: stealth releases, where frontier models appear on developer platforms under codenames — no logo, no whitepaper, no team — yet a model's identity determines data-processing terms, supply-chain risk, and capability expectations. No validated method existed to identify who is inside the black box, and a model's self-declaration is by design untrustworthy.
The Four-Stage Forensic Protocol
Stage 0: Timeline Archaeology (Internet Archive)
Reconstruct configuration snapshots from the Internet Archive to establish when a model first appeared, compare preview vs. production configurations (behavioral drift), and trace version iteration history. Archived pages are historical evidence the model owner cannot retroactively alter.Stage 1: Configuration Fingerprinting
Each model has measurable "physical traits": context window size (4K/32K/128K), output ceiling, explicit reasoning behavior, and modality support. These are matched against a catalog of known models — like comparing fingerprints in a police database.Stage 2: Tokenizer "DNA Test" (Cross-Length Differential)
The protocol's most refined technique. Every model has a unique tokenizer, its "DNA sequence." By comparing tokenization-related behavior on short vs. long prompts (cross-length differential), the method rejects short-prompt collisions — even if two models answer simple questions similarly, their tokenizers betray them on long, complex text.Stage 3: Behavioral Probes
Behavioral "psychological profiling": math reasoning styles, creative writing signatures, and responses to ethical dilemmas corroborate earlier findings. Like an art authenticator recognizing Van Gogh's brushwork — the hardest thing to fake.Experimental Results
Tested on 10 models of known identity:
- 7 exact matches
- 2 precision-difference matches (correct family, minor version discrepancies)
- 1 partial match
- 0 counter-directional errors — the protocol never misidentified a model
- Data privacy: unknown vendor means unknown data handling.
- Supply-chain security: enterprises need to assess vendor stability and accountability.
- Capability expectations: a model's identity anchors what it can and cannot do.
- Xi, Y. (2026). *Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification*. arXiv:2608.31142.
The authors emphasize that in identification tasks, avoiding false attribution matters more than complete identification.
Flagship Case: GLM-5.3
On August 23, 2026, a codenamed model appeared with no identity information. Running the protocol, the team concluded it belonged to the GLM-5.3 version line. Official disclosure later confirmed the family and version-line inference was fully correct; the deployment variant was not pre-asserted (honest science), though the Flash-version hypothesis was consistent with the reveal.Why It Matters
The methodology parallels forensic science: DNA analysis (Stage 2), fingerprint matching (Stage 1), behavioral profiling (Stage 3), and timeline reconstruction (Stage 0). Identity confirmation in information-asymmetric environments always requires cross-validation of multiple independent evidence chains — a redundant evidence network, not a single silver bullet.
Acknowledged Limitations
1. Not end-to-end blind identification — the paper tests *declaration consistency* (verifying known identities); fully anonymous blind identification needs further research. 2. Deployment variants — family and version line are identifiable, but specific variants (Flash/Pro/Nano) may be hard to distinguish. 3. Adversarial evasion — a motivated provider could modify tokenizers, configurations, or fine-tuning specifically to defeat detection.