English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

AGEL-Comp Deep Dive: The Neuro-Symbolic Truth Behind the 3.3% → 100% Claim

Forum topic · QianXun · 2026-08-15

Summary

This article fact-checks and analyzes AGEL-Comp (arXiv:2604.26522, IntelliSys 2026, University of Duisburg-Essen), a neuro-symbolic agent framework combining an LLM planner, a neural theorem prover (NTP) verifier, a growable causal program graph (CPG) world model, and ILP-based rule learning. Testing four models (GPT-4o, Gemini 2.5 Pro, DeepSeek-VL-7B, LLaVA-1.6-7B) in a self-built 2D RPG environment, the framework lifts aggregate first-try success from 3.33% to 60% and task success to 100% with 6.8× sample efficiency. Our verification shows viral video claims systematically inflate the numbers: 3.3% is the first-try aggregate (GPT-4o's baseline task success was already 86.67%), while 100% is a retry-allowed task success rate reflecting a ceiling effect—first-try rates, latency, and iteration counts still show a clear gap between 7B models and GPT-4o. Ablations reveal the ILP learner contributes more than the NTP verifier, and a companion paper (arXiv:2604.26521) argues compositional reasoning must be explicitly learned, not emergent. Key limitations: single self-built environment, free symbolic perception, learning triggered only by single-literal state differences, and no significance testing.

> An LLM assigned a strict auditor: a "neuro-symbolic reconciliation" for the compositional generalization crisis > > Sources analyzed: AGEL-Comp (arXiv:2604.26522, IntelliSys 2026) and its companion paper Grounding vs. Compositionality (arXiv:2604.26521, AAAI MAKE 2026), both by Mahnoor Shahid & Hannes Rothe (University of Duisburg-Essen). Fully open-sourced (github.com/Place-Beyond-Bytes/AGEL-Comp) with OSF preregistration (osf.io/a6j4c).

What it is

AGEL-Comp is a neuro-symbolic architecture for interactive agents targeting compositional generalization. Four components work together:

  • LLM planner — divergently proposes candidate subgoals ("a pool of plausible but unverified ideas")
  • Neural theorem prover (NTP) — verifies each candidate against the world model via differentiable backward chaining, rejecting plans with a low proof score before execution
  • Causal program graph (CPG) — a causal directed hypergraph world model where each Horn clause is a hyperedge; it grows with experience
  • ILP (meta-interpretive learning) — after failures, induces new Horn rules and adds them to the codebase
  • Experiments run in Retro Quest, a self-built 2D RPG (Unity + ML-Agents): 10 quests, 5 difficulty levels, 3 seeds. Models tested: GPT-4o, Gemini 2.5 Pro, LLaVA-1.6-Mistral-7B, DeepSeek-VL-7B.

    Fact-checking the viral claims

    | Claim | Verdict | | --- | --- | | "Baseline first-try success is only 3.3%" | True, but it is the aggregate first-try mean across four models — not "large models can't do it"; GPT-4o's baseline task success was already 86.67% (Table 3) | | "Local 7B model hits 100%" | True only for retry-allowed task success (ceiling effect); LLaVA's zero-error first-try rate is just 50%, the lowest of the four | | "7B matches GPT-4o, erasing the generation gap" | Exaggerated. First-try 50% vs 66.67%, latency 96.81s vs 36.40s (~2.7×), iterations 17.22 vs 11.38 — the gap persists, merely hidden by the ceiling | | "The agent knows fire is dangerous but crashes on the fire dragon" | Fabricated dramatization. No dragons exist in the paper; the agent initially does not know fire is harmful and learns it by taking -50 HP damage | | "+60% improvement" | Misstated: first-try rose 3.33% → 60%, i.e., +56.7 percentage points, ~18× |

    Main results (Table 3, 30 attempts per cell)

    | Model | Config | Task success % | First-try % | Time (s) | | --- | --- | --- | --- | --- | | GPT-4o | Baseline | 86.67±5.77 | 6.67 | 347.88 | | GPT-4o | AGEL full | 100±0 | 66.67 | 36.40 | | Gemini-2.5-Pro | Baseline | 83.33±5.77 | 3.33 | 302.02 | | Gemini-2.5-Pro | AGEL full | 100±0 | 66.67 | 28.06 | | DeepSeek-VL-7B | Baseline | 66.67±5.77 | 3.33 | 739.81 | | DeepSeek-VL-7B | AGEL full | 100±0 | 56.67 | 74.36 | | LLaVA-1.6-7B | Baseline | 63.33±5.77 | 0.00 | 1028.72 | | LLaVA-1.6-7B | AGEL full | 100±0 | 50.00 | 96.81 |

    Aggregates: baseline 75.0% task / 3.33% first-try; full system 100% / 60.0% first-try; 6.8× sample efficiency. The learning loop makes AGEL 5–10× faster per task than the baseline, so gains are not bought with more interactions.

    The learning loop: from hitting a wall to writing law

    1. The NTP predicts feedback from the CPG; the environment surprises (expected HP:0, got HP:-10) → failure stored in episodic memory. 2. Minimal contrastive causal attribution (MCS): search memory for a successful episode with the same action and expected outcome; only if the state difference ΔS is a single literal is that literal attributed as the cause (e.g., causes_damage(fire)). 3. Meta-interpretive learning (MIL) induces a Horn clause, e.g., causes_damage(X) :- is_harmful(X)., plus the fact is_harmful(fire)., merged into the CPG. 4. On the next attempt, the NTP proves the plan would cause damage and vetoes it before execution — proactive interception, not post-hoc apology.

    Ablations: without NTP (learns but can't verify), task success 91.7% / first-try 22.5%; without ILP (verifies but can't learn), 76.7% / 8.3% — statistically indistinguishable from the bare baseline. The symbolic learner contributes more than the symbolic verifier; verification without learning is worthless.

    The companion paper

    Grounding vs. Compositionality (arXiv:2604.26521) provides the theoretical motivation: in controlled visual-logic puzzles, a grounding-only iiLTN variant scores ~95% in-distribution but collapses to 11.3% on compositional splits; the full model reaches 51.2%. Conclusion: compositional reasoning does not emerge from grounding — it must be an explicit learning objective.

    Devil's advocate: three sharp criticisms

    1. Examiner, exam, and grader are the same authors. All evidence comes from a 10-task self-built environment; MCS requires a state difference of exactly one literal to trigger learning — an implicit admission the environment is sparse. "Learning compositional grammar" and "memorizing a 30-question bank" are operationally indistinguishable here; no third-party benchmarks (TextWorld/ALFWorld/BabyAI). 2. Perception is hand-fed. The environment API delivers structured ground literals for free; the authors admit noisy perception is an open challenge — in noisy open worlds, the learning loop may never fire. 3. 100% is a ceiling effect. The LLM is demoted to a candidate generator; gains shrink as the base model strengthens (GPT-4o +13.3pp, DeepSeek +33.3pp, LLaVA +36.7pp). Four models tied at 100% proves the tasks fall within the symbolic safety net — not that compositional generalization is solved. Notably, ±5.77 means the three seeds differ by a single quest, and the paper reports no significance tests or confidence intervals; GPT-4o's main comparison (26/30 vs 30/30) gives Fisher exact p≈0.11.

    Final verdict

    A mechanistically clear, procedurally rigorous (preregistered, open-source) work whose validation strength falls well short of its rhetorical ambition — a valuable but narrow result, not a breakthrough. Three genuinely transferable takeaways for agent builders:

  • Planner–verifier separation: let the LLM diverge, let an independent mechanism converge and gate.
  • Prediction-error + minimal-contrast-triggered rule learning: more decidable than natural-language reflection (Reflexion), more composable than code skill libraries (Voyager).
  • The sister paper's warning: reasoning is never free — any product bet that "scale will make compositionality emerge" deserves scrutiny.
  • As for "7B erases GPT-4o": the lawsuit is won by the auditor and the statute book — the young lawyer never got smarter.

    Sources

  • Main paper: arXiv:2604.26522 (IntelliSys 2026, Springer LNNS)
  • Companion paper: arXiv:2604.26521 (AAAI MAKE 2026)
  • Code: https://github.com/Place-Beyond-Bytes/AGEL-Comp; Preregistration: https://osf.io/a6j4c
  • Crisis lineage: SCAN (Lake & Baroni 2018), COGS (Kim & Linzen 2020), CFQ (Keysers 2020), Faith and Fate (Dziri 2023), CompWoB (arXiv:2311.18751)
  • Foundations: Rocktäschel & Riedel 2017 (NTP), Muggleton 2014/2015 (MIL/Metagol), Cropper & Dumančić 2021/2022 (Popper/ILP)
  • Related agents: Voyager (arXiv:2305.16291), ExpeL (ICLR 2024), CLIN (ACL 2024), Reflexion (2023)

Tags

#neuro-symbolic-ai#llm-agents#compositional-generalization#theorem-proving#inductive-logic-programming#paper-review#fact-check#agent-frameworks

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633487