RATED: When the Floor Starts Moving, Don't Let Your AI Step into the Void

At 3 a.m., a convenience store's lights go out half by half — this is the metaphor behind RATED (Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics), a CIKM 2026 paper by Chen and Liao of Texas Tech University (arXiv:2610.13889). The system asks one question beyond "semantic best match": can this tool actually work right now — and learns to route around failures mid-task, with no model retraining and only +0.172 ms of extra latency per selection.

The Problem: Your Navigation Led You into a Construction Zone

A large language model thinks; the tools around it — weather APIs, calendar plugins, email services, database interfaces — are its limbs reaching into the real world. Tool retrieval decides which of hundreds or thousands of tools an agent should call, today almost always via vector similarity over names and descriptions. But that ranking assumes a static tool directory, while real tool ecosystems get sick:

• APIs go down (timeouts, 503s)
• Authentication expires (rotated keys, 401s)
• Tools silently change (response schemas drift away from the docs)

The retriever knows none of this. The result: agents pick the semantically best tool that is currently broken, and the failure only surfaces at execution time. The paper's target is exactly this blind spot — not "which tool matches best," but "which tool works now and matches best."

EcoAgent: A Benchmark That Gets Sick

• 5,172 real tools aggregated from four open-source agent platforms: LangChain, CrewAI, LlamaIndex, and n8n
• 210,000+ query-tool pairs, split into EcoAgent-Syn (training) and EcoAgent-Real (evaluation)
• A reliability score (0–1) per tool from three weighted signals: name–description consistency, source trustworthiness, and documentation quality
• Tools scoring ≥ 0.8 are labeled reliable: 2,940 tools, or 56.8% — meaning over 40% of the real ecosystem fails a basic diligence check

Architecture: A Frozen Translator Plus a Gambler

Stage 1 — the translator. A DRAGON dual-encoder, fine-tuned with InfoNCE contrastive learning on EcoAgent-Syn, encodes queries and tool descriptions into a shared vector space. After fine-tuning it is frozen: new tools only need one encoding pass into the vector store, no retraining.

Stage 2 — the gambler. A lightweight contextual bandit re-ranks the top-K candidates using partial feedback: each execution returns success (1) or failure (0), but only for the tool actually chosen. Failures push a tool's reliability estimate down; consecutive successes push it up — a continuous, recoverable quantity, unlike a hard blacklist. No neural network is involved at this layer; it is a statistically convergent online learner that learns only at deployment time.

Why two stages? Semantic matching handles slow variables (a tool's purpose rarely changes); reliability adaptation handles fast variables (health changes daily). Freezing the former and iterating the latter decouples knowledge from trust.

Results

Offline retrieval (MRR): RATED scores 0.618 vs ToolLLM's API retriever at 0.406 (+52.2%) — inside a dense pool of 5,172 tools. Notably, the authors honestly report that pure offline retrievers (OpenAI Ada-002, SPLADE-v3) still beat RATED on static MRR.

Fault injection — where the floor moves: Agents run T = 10,000 sequential tool-selection steps; faults are randomly injected after step 2,000, and the post-fault window covers steps 2,001–5,000.

| Setting | ToolLLM retriever | RATED | |---|---|---| | 30% fault rate — faulty-tool selection | 86.2% | 2.9% | | 20% fault rate — faulty-tool selection | 79.6% | 2.5% |

After faults begin, the ToolLLM-based agent hits a broken tool roughly 8.6 times out of 10 — effectively unusable. RATED avoids nearly all of them using reliability evidence learned within a few steps, with no retraining, human intervention, or alerting. It is best in all dynamic settings, against baselines including TF-IDF, BM25, Ada-002, BGE-M3, GTE-Large, E5-Large-v2, vanilla DRAGON, SPLADE-v3, Health Check Blacklist, and Retry (k=3). The extra latency is +0.172 ms on a 3.395 ms baseline.

Stated Limitations

• Fault injection is simulated; real-world failures are correlated (one cloud outage takes down many tools at once) — untested
• Cold-start behavior for brand-new tools (no bandit history) is not validated
• The system has not yet been run inside a full agent runtime with nested, multi-step, interdependent tool calls

Key Numbers

| Metric | Value | |---|---| | Real tools aggregated | 5,172 (LangChain, CrewAI, LlamaIndex, n8n) | | Query-tool pairs | 210,000+ | | Reliable tools (score ≥ 0.8) | 2,940 (56.8%) | | Offline MRR | 0.618 (vs ToolLLM 0.406, +52.2%) | | Faulty-selection @ 30% faults | ToolLLM 86.2% → RATED 2.9% | | Faulty-selection @ 20% faults | ToolLLM 79.6% → RATED 2.5% | | Extra latency | +0.172 ms (baseline 3.395 ms) | | Experiment scale | T = 10,000 steps; faults at step 2,000; 3,000-step post-fault window |

Takeaway

Reliability is not a property of a tool but a state of a tool within a time window. Baking it into model weights means永远 chasing a world that has already moved on; a frozen semantic translator plus a fast online trust learner lets the system adapt at the speed of reality — for less than a fifth of a millisecond. Static leaderboard wins do not automatically translate into production reliability: sometimes the answer is not a smarter model, but a more awake one that acknowledges the floor is moving.

Reference

Chen & Liao (Texas Tech University). "RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics." CIKM 2026. arXiv:2610.13889, 2026-10-09.