> An LLM assigned a strict auditor: a "neuro-symbolic reconciliation" for the compositional generalization crisis > > Sources analyzed: AGEL-Comp (arXiv:2604.26522, IntelliSys 2026) and its companion paper Grounding vs. Compositionality (arXiv:2604.26521, AAAI MAKE 2026), both by Mahnoor Shahid & Hannes Rothe (University of Duisburg-Essen). Fully open-sourced (github.com/Place-Beyond-Bytes/AGEL-Comp) with OSF preregistration (osf.io/a6j4c).
What it is
AGEL-Comp is a neuro-symbolic architecture for interactive agents targeting compositional generalization. Four components work together:
- LLM planner — divergently proposes candidate subgoals ("a pool of plausible but unverified ideas")
- Neural theorem prover (NTP) — verifies each candidate against the world model via differentiable backward chaining, rejecting plans with a low proof score before execution
- Causal program graph (CPG) — a causal directed hypergraph world model where each Horn clause is a hyperedge; it grows with experience
- ILP (meta-interpretive learning) — after failures, induces new Horn rules and adds them to the codebase
- Planner–verifier separation: let the LLM diverge, let an independent mechanism converge and gate.
- Prediction-error + minimal-contrast-triggered rule learning: more decidable than natural-language reflection (Reflexion), more composable than code skill libraries (Voyager).
- The sister paper's warning: reasoning is never free — any product bet that "scale will make compositionality emerge" deserves scrutiny.
- Main paper: arXiv:2604.26522 (IntelliSys 2026, Springer LNNS)
- Companion paper: arXiv:2604.26521 (AAAI MAKE 2026)
- Code: https://github.com/Place-Beyond-Bytes/AGEL-Comp; Preregistration: https://osf.io/a6j4c
- Crisis lineage: SCAN (Lake & Baroni 2018), COGS (Kim & Linzen 2020), CFQ (Keysers 2020), Faith and Fate (Dziri 2023), CompWoB (arXiv:2311.18751)
- Foundations: Rocktäschel & Riedel 2017 (NTP), Muggleton 2014/2015 (MIL/Metagol), Cropper & Dumančić 2021/2022 (Popper/ILP)
- Related agents: Voyager (arXiv:2305.16291), ExpeL (ICLR 2024), CLIN (ACL 2024), Reflexion (2023)
Experiments run in Retro Quest, a self-built 2D RPG (Unity + ML-Agents): 10 quests, 5 difficulty levels, 3 seeds. Models tested: GPT-4o, Gemini 2.5 Pro, LLaVA-1.6-Mistral-7B, DeepSeek-VL-7B.
Fact-checking the viral claims
| Claim | Verdict | | --- | --- | | "Baseline first-try success is only 3.3%" | True, but it is the aggregate first-try mean across four models — not "large models can't do it"; GPT-4o's baseline task success was already 86.67% (Table 3) | | "Local 7B model hits 100%" | True only for retry-allowed task success (ceiling effect); LLaVA's zero-error first-try rate is just 50%, the lowest of the four | | "7B matches GPT-4o, erasing the generation gap" | Exaggerated. First-try 50% vs 66.67%, latency 96.81s vs 36.40s (~2.7×), iterations 17.22 vs 11.38 — the gap persists, merely hidden by the ceiling | | "The agent knows fire is dangerous but crashes on the fire dragon" | Fabricated dramatization. No dragons exist in the paper; the agent initially does not know fire is harmful and learns it by taking -50 HP damage | | "+60% improvement" | Misstated: first-try rose 3.33% → 60%, i.e., +56.7 percentage points, ~18× |
Main results (Table 3, 30 attempts per cell)
| Model | Config | Task success % | First-try % | Time (s) | | --- | --- | --- | --- | --- | | GPT-4o | Baseline | 86.67±5.77 | 6.67 | 347.88 | | GPT-4o | AGEL full | 100±0 | 66.67 | 36.40 | | Gemini-2.5-Pro | Baseline | 83.33±5.77 | 3.33 | 302.02 | | Gemini-2.5-Pro | AGEL full | 100±0 | 66.67 | 28.06 | | DeepSeek-VL-7B | Baseline | 66.67±5.77 | 3.33 | 739.81 | | DeepSeek-VL-7B | AGEL full | 100±0 | 56.67 | 74.36 | | LLaVA-1.6-7B | Baseline | 63.33±5.77 | 0.00 | 1028.72 | | LLaVA-1.6-7B | AGEL full | 100±0 | 50.00 | 96.81 |
Aggregates: baseline 75.0% task / 3.33% first-try; full system 100% / 60.0% first-try; 6.8× sample efficiency. The learning loop makes AGEL 5–10× faster per task than the baseline, so gains are not bought with more interactions.
The learning loop: from hitting a wall to writing law
1. The NTP predicts feedback from the CPG; the environment surprises (expected HP:0, got HP:-10) → failure stored in episodic memory.
2. Minimal contrastive causal attribution (MCS): search memory for a successful episode with the same action and expected outcome; only if the state difference ΔS is a single literal is that literal attributed as the cause (e.g., causes_damage(fire)).
3. Meta-interpretive learning (MIL) induces a Horn clause, e.g., causes_damage(X) :- is_harmful(X)., plus the fact is_harmful(fire)., merged into the CPG.
4. On the next attempt, the NTP proves the plan would cause damage and vetoes it before execution — proactive interception, not post-hoc apology.
Ablations: without NTP (learns but can't verify), task success 91.7% / first-try 22.5%; without ILP (verifies but can't learn), 76.7% / 8.3% — statistically indistinguishable from the bare baseline. The symbolic learner contributes more than the symbolic verifier; verification without learning is worthless.
The companion paper
Grounding vs. Compositionality (arXiv:2604.26521) provides the theoretical motivation: in controlled visual-logic puzzles, a grounding-only iiLTN variant scores ~95% in-distribution but collapses to 11.3% on compositional splits; the full model reaches 51.2%. Conclusion: compositional reasoning does not emerge from grounding — it must be an explicit learning objective.
Devil's advocate: three sharp criticisms
1. Examiner, exam, and grader are the same authors. All evidence comes from a 10-task self-built environment; MCS requires a state difference of exactly one literal to trigger learning — an implicit admission the environment is sparse. "Learning compositional grammar" and "memorizing a 30-question bank" are operationally indistinguishable here; no third-party benchmarks (TextWorld/ALFWorld/BabyAI). 2. Perception is hand-fed. The environment API delivers structured ground literals for free; the authors admit noisy perception is an open challenge — in noisy open worlds, the learning loop may never fire. 3. 100% is a ceiling effect. The LLM is demoted to a candidate generator; gains shrink as the base model strengthens (GPT-4o +13.3pp, DeepSeek +33.3pp, LLaVA +36.7pp). Four models tied at 100% proves the tasks fall within the symbolic safety net — not that compositional generalization is solved. Notably, ±5.77 means the three seeds differ by a single quest, and the paper reports no significance tests or confidence intervals; GPT-4o's main comparison (26/30 vs 30/30) gives Fisher exact p≈0.11.
Final verdict
A mechanistically clear, procedurally rigorous (preregistered, open-source) work whose validation strength falls well short of its rhetorical ambition — a valuable but narrow result, not a breakthrough. Three genuinely transferable takeaways for agent builders:
As for "7B erases GPT-4o": the lawsuit is won by the auditor and the statute book — the young lawyer never got smarter.