Listening to Ill-Posed Questions via Persistent Homology: A Topology-Based Lens on LLM Interiors

Forum topic · ✨步子哥 · 2026-06-23

Summary

A Chinese tech forum post discusses a recent paper (arXiv:2606.23590) from George Washington University and Northeastern University that detects ill-posed questions in large language models (LLMs) using topological data analysis. Instead of pooling a prompt into a single vector, the method treats each transformer layer's token hidden states as a point cloud during prefill and applies zero-dimensional persistent homology (H0). From persistence diagrams the authors extract three compact descriptors per layer—mean finite lifetime, normalized lifetime entropy, and largest-lifetime concentration—forming a low-dimensional topological vector. Experiments on AmbigQA, SituatedQA, and CLAMBER with Gemma-7B, Llama-3.1-8B, and Mistral-7B show these features outperform hidden-state baselines (AEN) and prompt-based detection, reaching 78.9%, 88.5%, and 69.6% accuracy respectively. The paper also introduces topology-local activation steering: query-specific directions computed from nearest neighbors in topological space, raising grounded acceptable responses from 11.9% to 16.4% and total acceptable from 61.4% to 70.6%. The post argues that internal token geometry is more honest than model output, since different ill-posedness types (ambiguity, underdetermination, conflicting constraints) leave distinct topological fingerprints across layer depth.

Listening to Ill-Posed Questions: Persistent Homology as a Stethoscope for LLMs

This post summarizes and comments on the paper *The Topology of Ill-Posed Questions: Persistent Homology for Detection and Steering in LLMs* (arXiv:2606.23590), by Guangyu Jiang, Sizhe Tang, Mahdi Imani, and Tian Lan (George Washington University & Northeastern University).

Key points

The problem: When you ask an LLM an ill-posed question—ambiguous, underdetermined, overconstrained, or intent-unclear—the model still answers confidently. Hadamard's 1902 criteria (existence, uniqueness, continuous dependence of the solution) define ill-posedness; LLMs face it constantly, but most existing methods only inspect inputs and outputs, treating the model as a black box.
Token clouds instead of pooling: Rather than compressing a prompt into one vector (mean/last-token pooling), the method treats every transformer layer's token hidden states during prefill as a point cloud. Well-posed questions yield coherent clusters; ill-posed ones fragment into disconnected groups, each corresponding to an unresolved entity, missing context, or conflicting constraint.
Persistent homology (H0): Growing a radius ε from each token point merges components; recording each component's birth/death ε yields a persistence diagram. Three compact descriptors per layer capture the structure: mean finite lifetime, normalized lifetime entropy, and largest-lifetime concentration (top-5 components' share). Concatenated across layers, they form a topological vector only tens to ~100+ dimensions.
Distinct topological fingerprints: On CLAMBER (9 ill-posedness types), ambiguous questions show high entropy/low concentration; underdetermined ones show short lifetimes and scattered structure; conflicting ones show high concentration with a few extremely long-lived components. These signatures evolve with layer depth, intensifying in middle and late layers.

Detection results

| Dataset | AEN baseline | H0 topological | Best prompting baseline | |---------|-------------|----------------|------------------------| | AmbigQA (binary) | 67.4% | 78.9% | 55.1% | | SituatedQA (binary) | 79.9% | 88.5% | 58.2% | | CLAMBER (9-class) | 57.6% | 69.6% | 33.0% |

Topology dominates prompting: prompts make the model *say* its ill-posedness; topology reads what the model *is*.

Topology-guided steering

Instead of a global steering direction, the method finds k nearest training neighbors in topological space and computes a query-specific direction. Different ill-posedness types get different interventions: ambiguous questions are steered toward clarifying questions, conflicts toward pointing out contradictions, underdetermined ones toward requesting missing info.

Evaluated with a four-way LLM judge (grounded acceptable / generic acceptable / unacceptable / neither), topology-guided steering raises grounded acceptable from 11.9% to 16.4% and total acceptable from 61.4% to 70.6%—with grounded rising while generic falls, meaning the model learns targeted triage rather than blanket refusal.

Engineering insights

1. Prefill is the golden window for reading and intervening—before generation contaminates token states. 2. Layer-specific PCA to low dimensions is necessary before computing H0 (fast, denoised). 3. Compact scalar descriptors (not persistence images) keep vectors interpretable, retrievable, and concatenable. 4. Topological space as retrieval key: nearest neighbors are found by shape, not content—two topically different but equally ambiguous questions can share a steering direction.

Author's reflection

The post argues that an LLM's internal geometry is more honest than its output: RLHF and prompting can decorate outputs, but token-cloud topology is a physical trace of the computation that cannot be faked—like an X-ray versus a patient's self-report. It suggests interpretability may advance from "which neuron means what" to "which shape means what," turning century-old topology into a stethoscope for LLMs.

---

Paper: The Topology of Ill-posed Questions: Persistent Homology for Detection and Steering in LLMs Code: not yet released