Slot Machines: How LLMs Keep Track of Multiple Entities in Their Heads
A Simple Test
"Alice is tall. Bob is short."
As a human, you effortlessly retain two facts: Alice is tall, Bob is short. Asked "Who is tall?" you instantly answer "Alice."
This is trivial for humans. But for large language models (LLMs), the question is far more complex than it appears.
An LLM processes one token at a time. By the time it reads "Bob," the information "Alice is tall" has already "passed by." How does the model remember it? More fundamentally: when multiple entities appear in the context simultaneously, how does the model track them all in its "head"?
In April 2026, Anthropic researchers Paul C. Bogdan and Jack Lindsey published the paper *Slot Machines: How LLMs Keep Track of Multiple Entities*, using an elegant probing method to reveal the mechanism by which LLMs internally represent multiple entities. The answer is surprising yet sensible.
The "Slot Machine" Metaphor
The title "Slot Machines" is a clever pun. In computer science, a "slot" is a structured space for storing information; a slot machine is also, of course, the gambling device. The authors use the double meaning to describe both the internal "slot" mechanism and its somewhat "gambly" nature (explained later).
The core finding: at every token position, the model maintains two nearly orthogonal "slots" — a current-entity slot and a prior-entity slot.
Two Slots, Two Functions
Consider the text: "Alice is tall. Bob is short."
When the model processes "Alice is tall," Alice's and "tall" information is encoded in the current-entity slot.
When processing "Bob is short," two things happen: 1. Bob's and "short" information is encoded in a new current-entity slot 2. Alice's and "tall" information is copied into the prior-entity slot
In other words, while processing "Bob," the model simultaneously stores two entities in two slots — like a two-compartment box: the front holds the current item, the back holds the previous one. Each new item pushes the old one back.
But the Two Slots Serve Different Purposes
This is one of the paper's most interesting findings.
The current-entity slot answers factual questions about specific entities — "Who is tall?", "What is Bob's trait?" — by querying the current-entity slot at the corresponding entity token position.
The prior-entity slot supports relational reasoning — "Who comes after Alice?", "Do Alice's and Bob's traits conflict?" — questions requiring comparison of adjacent entities.
Through careful multi-slot probing experiments, the researchers trained linear probes to decode both the current and prior entity from a single token's residual stream activations, confirming the two pieces of information live in nearly orthogonal subspaces.
A Puzzling "Information-Usage Gap"
Here is a particularly striking discovery: even when information is linearly decodable from the prior-entity slot, the model does not use it when answering factual questions.
Analogy: you put a document in a drawer (the prior-entity slot). The information is genuinely there — but when you need to answer a question, you never open that drawer; you only search the bookshelf (the current-entity slot).
This reveals a deep "information-usage gap": information available inside the model and information the model actually uses are two different things.
"Alice Prepares and Bob Consumes Food" — the Model's Blind Spot
This gap shows up dramatically in a special grammatical construction.
Consider: "Alice prepares and Bob consumes food."
The peculiarity: the verbs "prepares" and "consumes" share the same object "food," but have different subjects (Alice and Bob). At the token "food," the model must simultaneously bind two subject-verb-object relations.
The researchers found that open-source models almost entirely fail on this construction — accuracy near random guessing.
The reason is intuitive: the current dual-slot mechanism can track only two entities, but this sentence requires maintaining two complete SVO bindings at the same token, exceeding the dual-slot capacity.
Interestingly, latest frontier models (such as Claude) handle this structure correctly, suggesting they may have developed more complex entity-binding strategies — perhaps more than two "compartments."
Connections to Sycophancy and Deception
In the discussion, the authors make a bold speculation: the current/prior-entity dual-slot structure may be a natural substrate for sycophantic and deceptive behavior in models.
Why? Because both behaviors require the model to hold two different "perspectives" or "representations" simultaneously:
- Sycophancy: the model needs to know both "the answer the user wants" and "the correct answer," then choose to output the former — like storing "facts" in one slot and "flattery" in the other.
- Deception: the model needs to know both "the truth" and "what it wants you to believe."
- Probing experiments mostly use synthetic text, which differs from real conversational settings
- Whether the "dual-slot" conclusion holds across all model scales and architectures needs further validation
- The connection to sycophancy/deception is only speculative, lacking direct evidence
The dual-slot mechanism provides exactly this capacity for "holding two pieces of information at once." Of course, this is only speculative — the paper does not directly prove it — but it offers an illuminating new lens on model misbehavior.
An Honest Assessment
Highlights of the paper:
Elegant method. The multi-slot probing approach, decoding multiple entities from a single token's activations, is rare in prior mechanistic interpretability research.
Novel findings. The information-usage gap is especially valuable — it reminds us that information merely existing inside the model does not mean the model uses it, with important implications for understanding model reasoning.
Broad connections. Linking entity tracking to sycophancy and deception is speculative but provides a thought-provoking framework.
Limitations abound:
Next time you chat with an AI, consider: at the instant each word you type is processed, the model's "mind" may be busily shuttling information between two slots — one for "now," one for "just before."
---
Paper link: arXiv:2604.21139