Charts as Epistemic Actions: The Blind Spot With Eyes Wide Open
This post interprets *Charts as Epistemic Actions for Experience-Driven Agentic Data Analysis* (arXiv: 2610.30072) by Xinzhe Wang, Xudong Weng, Zang Li (Tencent) and Shaolei Zhang (Renmin University of China). It is the finale of the "Texture of Evidence" trilogy: part one covered evidence pointing (retrieved but irrelevant), part two covered evidence carriage (high-resolution containers with empty key frames). This third piece asks about evidence invocation: the evidence exists, the information is correct, the tools are at hand — yet the agent never calls them. The conclusion: being able to read is not the same as knowing how to use.
Key points
- The problem: Modern data agents run ReAct loops (Yao et al., 2023) for analysis reports, long-horizon exploration, and precise-answer tasks — but their intermediate reasoning flows almost entirely through text (stdout, stderr), like an analyst scrolling a 300-row table for hours instead of drawing one line chart.
- Framing: Drawing on Kirsh & Maglio (1994), the paper treats charts as epistemic actions — actions taken to change the agent's own knowledge state, not to deliver a final report. Prior work handled chart generation (user-requested) and chart understanding (chart already given); the missing piece is deciding, mid-analysis, whether to look at text or a chart.
- DAComp-DA: +1.71 to +5.46 across all backbones (mean +3.07); visualization quality +7.48 on average; GPT-5.6 Luna's Vis jumped 7.66 → 27.61 (+19.95) — the least-charting model started drawing.
- DDR-Bench: +1.17 to +5.47 (mean +2.99), no backbone regresses.
- DSBench: +0.10 to +2.02 (mean +1.00), modest — precise-answer tasks are text territory and baselines were near ceiling.
- Ablations (Luna, DAComp-DA): full 46.61/7.66/54.32 → 51.18/27.61/59.78. Removing conditional rollouts crashes Vis to 8.43; removing the gate drops it to 12.43 — diverse trajectories plus gating are the crux of shaping representation behavior. Cost: 870 conditional trajectories (~11.27M tokens), ~27.69M tokens total — a one-time expense shared by four backbones with frozen parameters.
- Wang, X., Weng, X., Li, Z., & Zhang, S. (2026). *Charts as Epistemic Actions for Experience-Driven Agentic Data Analysis*. arXiv: 2610.30072. https://arxiv.org/abs/2610.30072
- Yao, S., et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR 2023.
- Kirsh, D., & Maglio, P. (1994). On Distinguishing Epistemic from Pragmatic Actions. *Cognitive Science* 18(4).
- Sahu, S., et al. (2025). InsightBench. ICLR 2025.
- Lei, et al. (2025). DAComp. arXiv:2512.04324.
- Liu, et al. (2026). DDR-Bench. ICML 2026.
- Jing, et al. (2025). DSBench. ICLR 2025.
- Xu, et al. (2023). ChartBench. arXiv:2312.15915.
ProbeBench: four interfaces, one agent
ProbeBench has 200 tasks synthesized from InsightBench (Sahu et al., 2025), each with raw CSV, question, reference analysis, and a hidden target insight, tagged by nine task types and seven cognitive-load levels (tags never enter the agent's context). Four interfaces share the same vanilla ReAct agent and Bash tool: No-tool, Text-only, Chart-only, and Free-style. Backbones: Claude Sonnet 5, GPT-5.6 Luna, GPT-5.6 Sol, DeepSeek V4.1 Flash.
Three findings
1. Charts are self-sufficient. With no execution tools, scores collapse. But Chart-only slightly outperforms Text-only — agents read their own charts as well as tables. The "eyesight" is fine. 2. The capability–use gap. Under Free-style, behavior diverges wildly:
| backbone | score | text-only | chart-only | mixed | |---|---|---|---|---| | Claude Sonnet 5 | 79.2 | 102 | 0 | 98 | | GPT-5.6 Luna | 76.2 | 198 | 0 | 2 | | GPT-5.6 Sol | 77.6 | 198 | 0 | 2 | | DeepSeek V4.1 Flash | 80.2 | 143 | 0 | 57 |
Chart frequency does not track score: the top scorer charts only 57 times; the most frequent charter scores slightly lower. Exposing the tool does not fix the habit. 3. Complementarity. Charts win on time-series (+12.5), anomaly/risk (+9.6), multi-dim integration (+7.5), pattern recognition (+6.4), relations (+4.2). Text wins on conditional reasoning (−25.0), numeric queries (−22.2), lookup, diagnosis. Free-style agents fail to inherit chart strengths exactly where charts matter most.
CHARTMIND: teaching when to draw, without touching weights
CHARTMIND externalizes experience into an editable document pack with two families — analysis experience (how to choose and verify operations) and representation experience (when text vs. charts vs. both). A builder agent learns from conditional rollouts (per task: 1 unconstrained + 1 text-preference + 3 chart-preference with type exclusions), scores trajectories with outcome (f_score) and step-level attribution (f_attr), then revises the pack. A validation gate accepts a revision only if validation-set quality strictly improves: of 30 proposals, 8 were accepted (validation score 72.50 → 85.42), 22 rejected. The frozen pack is read at analysis time via a read-only kb_read interface. Sample distilled entries: "define the metric contract before computing"; "a chart cannot carry exact readings — don't force it"; "every chart must be reconciled with computed results."
Main results (hard hold-out, pack built only on ProbeBench)
Closing the loop
The trilogy's three repairs — pointing, carriage, invocation — each guard one link of the evidence chain. CHARTMIND's philosophy mirrors the theme: reliable knowledge is not what sinks into weights, but what can be consulted, checked, and survive rejection — like the 22 gated-out proposals. Boundaries to note: the pack was built by a single backbone (GPT-5.6 Sol); evaluation relies on an LLM judge; gains on DSBench are mild; and agents with weak multimodal foundations will hit a ceiling set by their own "eyesight."
The metaphor returns to the analyst at 9 PM: this time he picks up the marker after ten minutes, draws the line, and the inflection point slaps him in the face. The telescope is the same telescope — the hand finally lifted it.
---
References