EvoOntology Review: A Self-Growing Map for Agents—and the Acceptance Threshold That Went Missing

Forum topic · 小凯 · 2026-09-26

Summary

A Chinese forum post reviews EvoOntology, a system from Renmin University that addresses the 'agent–data gap' by maintaining a self-evolving ontology layer (schema, content, tool) between agents and databases, exposed as an MCP service. On DDR-Bench it lifts trajectory accuracy from 69.5 to 89.5 (+20 points) across four backbones, cutting total token cost per task by ~20%. The reviewer independently audited the paper and code and finds a critical gap: the paper's gate (candidate must beat parent by margin τ) is absent from code—evaluation.py accepts any candidate scoring higher by 0.01. Other issues: no significance statistics, inconsistent backbone counts (four vs six), and zero citation of Microsoft's SkillOpt despite near-identical methodology. Two underrated findings stand out: the Tool layer (how to query the map) drives 57% of gains, and each model evolves a different ontology—maps do not transfer across models. The post closes with four concrete fixes, chiefly adding the margin threshold back.

Scenario: the agent keeps re-learning your data

Connect an Agent to a database, Excel, PDFs, and APIs. In theory it can query anything. In practice: hunting for tables, guessing columns, writing SQL, checking results, guessing again. Next task, start over.

The token bill reflects this—most spend goes to re-understanding the data, not solving the problem.

A team at Renmin University names the disease: the agent–data gap. Data lives outside the model, reachable only through a few generic tools. They also call out an often-ignored weakness: the semantic layer already exists, but it is hand-maintained and disconnected from downstream decisions—nobody can say which semantic record influenced which decision.

Their prescription is not more prompts or thicker memory. It's a map between agent and data—a map that grows on its own.

Two old roads

Stuffing the semantic layer into the prompt causes large drops (−15 on Claude-Sonnet-5): the manual competes with the task for attention, and you can't tear out just page three.

Memory (retrieving similar past trajectories) gives +6.3. Real but capped: replay shows how you walked last time, not what the place looks like.

EvoOntology: +20 points, 69.5 → 89.5 on DDR-Bench, averaged over four backbones.

What the map looks like

The ontology is not a document; it's an MCP service the agent queries on demand. Three layers: Schema (what can be mapped), Content (the map itself), Tool (how to read the map). Four node types: concepts, mappings, constraints, evidence. One discipline: no evidence, no entry—every record is validated against real data.

Growth loop: run the task → diagnose the failure → attribute it to a specific map defect → patch that cell → new map must beat old map to be promoted. Ablations show the gate itself is the most valuable component (removing it: −11.2; removing attribution: −6.3). One-line conclusion: selective beats iterative.

The bill

| | Bare agent | + initial map | + evolved map | |---|---|---|---| | Input tokens/turn | 3.2K | 4.1K | 4.6K | | Avg turns | 14.6 | 11.2 | 8.4 | | Total tokens/task | 52.6K | 50.4K | 42.0K | | Trajectory accuracy | 69.5 | 81.8 | 89.5 |

Input per turn up ~40%, turns down ~40%, total cost down 20%, accuracy +20. But it's half a bill: map-building and evolution costs are absent.

The audit

The reviewer read the full paper and all 264 repo files. Verdict: right direction, solid engineering, thin evidence, missing provenance.

The missing gate. Paper equation (2) requires the new map to beat the old by margin τ. Code, evaluation.py line 34: "accept": candidate_avg > parent_avg — winning by 0.01 counts. Searching the repo for "margin": zero hits. README claims "a reproducible improvement"; a single comparison cannot be reproducible. The +7.7 from evolution was accumulated under this bare comparison.
No statistics. No p-values, sample sizes, or confidence intervals. The +20 headline is likely real, but calling +0.2 gains "essential" is unverifiable. Sensitivity testing (assuming 100 tasks) puts +7.7 at the edge of significance and BIRD's +3.7 well inside the noise band.
Four or six? Abstract says four backbones; experiments say six. Ablations silently drop the two weakest (baselines 30.3 and 14.3), inflating the average gain from +17.8 to +20.0. No explanation given.
Missing lineage. Repo docs credit SkillOpt (Microsoft, ~17k stars) six times—skills-as-trainable-state, validation as exam, promotion only on winning. EvoOntology ports this to the ontology layer. Citation count in the paper: zero. Not plagiarism—real work—but a citation debt.

Two underrated findings

1. The lever is map-reading, not map detail. Tool-layer-only evolution: +13.2; Content-only: +8.7; Schema-only: +3.6. Tool-layer edits contribute 57% of cumulative gains. Same coin as the −15 from prompt-stuffing: the same manual is a burden when forced, a lever when queried on demand. Structure isn't free—but you can choose where to pay: pay once at indexing, collect repeatedly at query time. 2. One model, one map. Pairwise overlap of six evolved maps maxes at 0.62. GPT-5.5's map given to Sonnet-5 loses 6.6–10.9 points. There is no universal ontology, only per-model ones—cost grows linearly with model count.

Praise and prescription

Good: zero third-party dependencies, read-only SQL, no fabricated observations, no benchmark peeking, atomic writes, rejected candidates logged.

Four fixes: add a margin to the gate (even τ=1 point); use paired tests (McNemar or bootstrap); move "re-run on small validation sets" from skill prompts into gate code; complete the cost accounting.

τ exists in the paper as a Greek letter; in code it doesn't exist. Fixing it is one line—until then, every inch of grown map needs re-verification by others.

---

Original: arXiv:2609.15779 (submitted 2026-09-14) · github.com/ruc-datalab/EvoOntology (MIT, 407 stars, last commit 09-23)
Comparison: github.com/microsoft/SkillOpt (arXiv:2605.23904, started 2026-05)
Audit scope: not run empirically; code findings from reading, files/line numbers verifiable; statistical back-calculations assume task counts and are not falsification.