When the Last Fruit Falls: Signatures of Semantic Search in LLM Activations
This post is a Feynman-style walkthrough of Leckie, Todd & Foster (2026), "Signatures of semantic search in the activations of large language models" (arXiv:2609.35599). Below is a structured English summary of the full argument.
Background: the semantic fluency task and optimal foraging
- The semantic fluency task (SFT) — "name as many animals as you can in one minute" — has been studied for decades. Humans don't produce words randomly: output comes in clusters (lions, tigers, leopards...) separated by switches (whale!, dolphin, shark...), a dual-component structure described by Troyer, Moscovitch & Winocur (1997).
- Hills, Jones & Todd (2012) reframed this as optimal foraging in semantic memory, borrowing Charnov's (1976) marginal value theorem: leave a patch when its local yield drops below the environment's average rate — like a fruit-picker abandoning a tree once only high branches remain.
- Human evidence: longer response latencies before switches; fMRI (Lundin et al. 2023) showing posterior cerebellum/hippocampus activation that ramps within a cluster like a depletion timer; disrupted hippocampal ripples in schizophrenia (Nour et al. 2023); and category-level priming (Neely 1977).
- The J-lens (from Gurnee et al., 2026) precomputes a Jacobian map that translates any intermediate residual-stream state back into vocabulary-space activation scores — a per-layer "teleprompter monitor."
- The J-space is the top-25 activated words at each step, proposed as an analogue of a global workspace.
- Probability dip: animals involved in switches had significantly lower next-token probability before being emitted, in all five models (χ² ≥ 32.39, p < 0.0001) — the machine's "hesitation before jumping patches."
- Mid-layer signature: from layer 8 upward, switch-word activations rose less than cluster-word activations, strengthening toward the final layers (χ² ≥ 98.17, p < 0.0001).
- Depletion predicts leaving: the number of not-yet-said animals from the current category available in J-space predicted switching at r ≈ −0.31 to −0.50 in mid layers, outperforming simple counts of the global word pool (F ≥ 19.94) — the model decides from locally available estimates, exactly as the marginal value theorem prescribes.
- Category pre-activation: abstract category labels (e.g., "water") ramp up *before* a switch into that category, peaking at the switch moment (F ≥ 848.35, p < 0.0001); competing category labels rise, then collapse at the switch — a winner-take-all race.
- Category steering vectors (item-level averages or abstract label vectors) added to the residual stream causally pulled models into target categories — e.g., Gemma-2-9B reached a 0.68 target-switch probability at layer 40, with category-level steering most effective in mid-to-late layers.
- A general "switch direction" (decoded via PCA + logistic regression, category-disjoint train/test splits) was detectable at switch events (AUC up to 0.87 at layer 16) and even prospectively (AUC 0.72), with within-cluster ramping along this direction (χ² ≥ 20.44, p < 0.0001) — a mechanical mirror of the hippocampal depletion timer.
- Steering along this direction pushed Gemma-2-9B's switch rate from a 0.29 baseline to 0.70 (positive) or down to 0.07 (negative) at layer 26 (Fisher exact p < 0.0001 vs. orthogonalized noise controls).
- The full foraging pipeline — depletion signal, category pre-activation, competitive selection, causal steering — exists in LLM activations, suggesting explore–exploit dynamics may be a convergent solution for any autonomous system searching structured resource spaces, not a carbon-based specialty.
- The J-space competition parallels Global Workspace Theory's "election" of conscious content; the read-and-intervene toolkit offers functional-integration evidence relevant to debates about machine minds.
- Applications could include tuning semantic rhythm and coverage in narrative generation, and search-style behaviors (web retrieval, tool use, long-context fact finding).
- Largest model is only 9B parameters; frontier models may differ.
- SFT is a controlled microcosm; switches in real dialogue and long narratives are unverified.
- Negative steering was weak in some models, and layer selectivity is not fully explained.
The experiments
Five open instruction-tuned models — Gemma-2-9B (42 layers), Gemma-2-2B (25), Llama-3.1-8B (32), Llama-3.2-3B (28), Qwen2.5-7B (28) — were prompted to name animals comma-separated, with nucleus sampling (temperature 0.9, top-p 0.95), repeated over 100 random seeds per model. Switches were defined strictly: consecutive animals sharing no Troyer-category labels. Models produced 36–65 animals per list with average cluster lengths of 2.2–3.2 — human-like clustering at the behavioral level.