SMART: State-Based Multi-Agent Reasoning Traces for Verification-Oriented Problem-Solving
> Paper: SMART: State-based Multi Agent Reasoning Traces for Verification-oriented Problem-Solving > arXiv: 2611.08196 | Authors: Sai Vallurupalli, Francis Ferraro / University of Maryland Baltimore County > Posted: 2026-11-05
The Problem: Reasoning Needs Knowledge Not Present in the Question
SMART addresses claim validation: given a scientific claim (e.g., "JAM-A knockout decreases intestinal permeability"), decide whether it is supported or refuted, with a confidence score and rationale. The required knowledge is often implicit—correctly judging the JAM-A claim requires introducing the unstated entity "intestinal epithelial cells" and the intermediate concept of paracellular transport.
Two traditional approaches have systematic flaws:
- Chain-of-Thought (CoT): when the model lacks knowledge, it frequently refutes claims by citing "lack of supporting information"—logically unsound, since absence of support is not evidence of falsehood. On SciFact, CoT used this reasoning path on 23–78% of refuted claims.
- RAG: retrieval results are noisy and often mislead the model; RAG performed far below CoT in the paper's experiments.
- Datasets: SFBench (183 claims, multilevel labels), SciFact (188), SciFact-Open (206); 577 total.
- Models: Qwen3 (1.7B/8B/32B, thinking and non-thinking), Llama-3.1 8B/70B Instruct, GPT-5.4-mini, Claude Haiku 4.5; Gemma 3 (1B/4B/12B) underperformed across all configurations.
- Metric: Balanced Accuracy (BAccuracy), averaged over 3 runs.
- Configurations: SMART-PK, SMART-WS, X-SMART-PK (cross-model), vs. CoT and RAG baselines.
- Gains: SMART-PK improves BAccuracy by ~10% on SFBench and ~5% on SciFact vs. CoT, with peaks of 20%. RAG underperforms CoT; SMART-WS underperforms SMART-PK due to noisy search results.
- Better rationales: "lack of supporting information" refutations drop from 23–78% (CoT) to 11–46% (SMART). Under a "Refute Filter" LLM judge, CoT's BAccuracy inflated by 3.5–46.3% vs. only 0.6–26.1% for SMART-PK.
- Contrastive premises matter: using both sides of premise pairs yields 3–6% gains over single-sided premises, encouraging counterfactual reasoning. State-transition tokens performed comparably to natural-language premises and far better than reasoning graphs.
- CoRE module: filtering entities to those matching claim ordering beat full SMART in 12 of 17 dataset×model combinations.
- Cross-model synergy: all models performed best using premises curated by *other* models (e.g., Qwen3 8B or GPT-5.4-mini). GPT-5.4-mini gained 6.2% on SciFact using Qwen3 8B premises—premise generation and curation are distinct capabilities, enabling labor division.
- Scale isn't everything: Llama-3.1 70B underperformed Qwen3 32B due to weaker opposition-premise generation and curation.
- Vallurupalli, S., & Ferraro, F. (2026). SMART: State-based Multi Agent Reasoning Traces for Verification-oriented Problem-Solving. *arXiv:2611.08196*.
- Wadden, D., et al. (2020). Fact or Fiction: Verifying Scientific Claims. *EMNLP*. (SciFact)
- Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. *NeurIPS*.
- Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. *arXiv:2203.11171*.
- Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. *NeurIPS*.
SMART's idea: instead of direct reasoning or blind search, structurally acquire knowledge about the claim's entities, verify that knowledge, then reason with verified premises.
The Four-Stage Framework
1. Identify: extract up to 5 key entities (including implicit ones) and assign each an expert LLM agent. Entities are treated as having modifiable states (e.g., JAM-A: expressed/impaired; tight junctions: intact/damaged). 2. Explore: each agent iteratively generates contrastive premise pairs—opposing statements about its entity—forcing consideration of multiple possible states rather than premature answers. The paper sets n×k = 10 premise pairs per claim (n iterations × k agents), with context injection so agents can build cross-entity links. 3. Curate: verify each premise's truth via either parametric knowledge (PK) self-reflection or web search (WS) with DuckDuckGo + NLI. Premises are also labeled by stance (support/oppose/unsure) with a 1–10 confidence score; only true, stance-relevant, high-confidence premises survive. 4. Infer: a meta-reasoner collectively weighs the support set P_s and oppose set P_o. Among five strategies tested (majority voting, reasoning judge, aggregating judge, critical judge, supporting judge), the Reasoning Judge—collective reasoning over all premises—performed best, beating majority voting by 8% (SFBench) and 5% (SciFact).
Unlike CoT, every reasoning step cites a concrete premise, making the output auditable—like a lawyer presenting evidence rather than thinking aloud.
Experiments: Three Datasets, Ten Models
Key Findings
Honest Limitations
1. Compute cost: ~10–20 API calls per claim; unsuitable for real-time use. 2. Web search non-reproducibility: SMART-WS results change over time; cached evidence snapshots recommended for critical uses. 3. Dependence on parametric knowledge: SMART-PK only extracts what the model already learned; long-tail knowledge may fail. 4. Small-model curation bottleneck: Qwen3 1.7B struggled with premise curation—use a larger model for curation and a smaller one for inference (the X-SMART approach). 5. Interpretability is graded, not binary: even SMART shows 0.6–26.1% performance inflation under the Refute Filter. 6. vs. ReConcile and KnowledgeNavigator: SMART outperforms both in most configurations; its edge comes from structured knowledge acquisition + verification, not merely multi-agent form.
The Bigger Picture: Precision-Repair Philosophy
The post frames SMART as the third in a trilogy: StepBack optimizes *reasoning* efficiency (repair only the faulty step), Compressed Memory optimizes *storage* efficiency (preserve both local and global fidelity), and SMART optimizes *knowledge* efficiency (structure, verify, then reason). The shared principle: rather than piling on compute, spend every unit of computation where it matters most—answering not just "can the system get it right," but "with how many resources."