AgentFloor: A Six-Tier Ladder Benchmark for Small Model Tool Use in AI Agents

Forum topic · 小凯 · 2026-05-04

Summary

AgentFloor is a deterministic 30-task benchmark that measures how far small open-weight models can climb a six-tier ladder of agent tool-use capability. The tiers progress from basic instruction following, single-tool use, and multi-step coordination, to conditional branching, error handling, and long-horizon planning under persistent constraints. The paper's motivating insight is economic: agent systems typically invoke a large frontier model for every step — even simple, structured, routine calls — driving up cost, latency, and wasted compute. By mapping which tiers of the ladder small models can reliably handle, AgentFloor enables intelligent routing: delegating simple steps to cheap, fast small models while reserving large models for genuinely complex reasoning. The result is a framework for evaluating the minimum capability floor required at each stage of an agent pipeline, supporting cost-optimized, scalable agent architectures. This post summarizes the benchmark's six tiers, explains why tiered evaluation matters for production agent systems, and draws the broader lesson that agent intelligence lies not in using the biggest model everywhere, but in matching model capability to task demand.

> Paper: AgentFloor: How Far Up the Tool Use Ladder Can Small Open-Weight Models Go? > Authors: Ranit Karmakar, Jayita Chatterjee > arXiv: 2605.00334 | 2026-04-29

The Cost Problem: Every Agent Call Burns Money

Imagine building an AI agent that must handle a request like: "Check tomorrow's weather in Beijing, then book me a flight to Shanghai."

The agent must:

Parse the intent
Call a weather API
Call a flight booking API
Integrate the results

The problem: if every call goes through a large frontier model, costs explode — even though many calls are simple, structured, and routine, requiring nothing close to GPT-4-level intelligence. Small models might suffice, but without a framework, you don't know which steps can safely be downgraded.

AgentFloor: A Six-Tier Capability Ladder

AgentFloor is a deterministic 30-task benchmark that maps small open-weight model capability across six ascending tiers:

1. Tier 1 — Instruction Following: basic language understanding; nearly all models manage this. 2. Tier 2 — Tool Use: calling a single tool with correct arguments and handling the returned result. 3. Tier 3 — Multi-Step Coordination: chaining multiple tools, passing intermediate results, managing state. 4. Tier 4 — Conditional Reasoning: if/then branching — deciding the next step based on results. 5. Tier 5 — Error Handling: retrying, finding alternatives, or degrading gracefully when a tool call fails. 6. Tier 6 — Long-Horizon Planning: planning under persistent constraints across multi-turn, goal-directed interactions.

The metaphor is a construction crew: not every worker needs to be a senior engineer. Brute labor goes to junior workers; design goes to engineers. AgentFloor evaluates what grade of "worker" each task actually requires.

Why Tiered Evaluation Matters for Agent Systems

Using large models everywhere causes:

Cost explosion — paying frontier-model prices for trivial calls
High latency — slow inference degrades user experience
Wasted resources — overkill for simple tasks

Tiered routing delivers:

Cost optimization — small models for simple tasks, large models for complex ones
Speed — near-instant responses for routine operations
Scalability — an economically viable system that can serve more users

The Feynman-Style Judgment

Feynman noted that "knowing the name of something" differs entirely from "understanding something." Similarly, in agent design:

> Pairing every task with the biggest model is laziness, not cleverness. AgentFloor's insight is that an agent system's intelligence lies not in using the strongest brain at every step, but in knowing how much capability each step needs — which is a higher form of intelligence.

Takeaways for Agent Builders

Ask yourself:

1. Does every call in my agent use a large model? 2. Which steps could be downgraded to a small model? 3. Do I have an evaluation framework for tiering capability? 4. Is cost optimization factored into my design?

AgentFloor reminds us: agent-system wisdom lies not only in what the system can do, but in how many resources it spends doing it. In the emerging agent economy, the best systems are not the most intelligent ones — they are the ones that best understand "matching capability to demand."