Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Reusable Task-Specific Solutions?

Forum topic · 小凯 · 2026-10-08

Summary

A new paper (arXiv:2610.08775) by Ankit Sonthalia, Haritz Puerto, Alexander Rubinstein, Martin Gubri, and Seong Joon Oh introduces "bottling" — the ability of LLM agents to convert general capabilities into task-specific solutions that balance answer quality against amortised cost. The authors present BOTTLED, a benchmark where agents receive an entire unlabelled workload and must complete it under fixed time, compute, and LLM API budgets, freely choosing their approach (e.g., training a small model or writing a reusable program). Experiments across ten models and three tasks show that strong zero-shot task performance does not reliably translate into strong bottling ability: of 60 bottling runs, 48 scored below the lower bound of their model's zero-shot 95% confidence interval. However, successful bottling can yield substantial savings — on a query-product relevance classification task, Opus 5 retained about 82% of its zero-shot macro-F1 at roughly 657x lower reported cost, and bottling proved competitive with Jev, a purpose-built "System-1" model for cheap repeated inference. BOTTLED lays a foundation for evaluating and improving how agents invest limited resources into reusable solutions for large, repetitive workloads.

Overview

Field: Machine Learning Authors: Ankit Sonthalia, Haritz Puerto, Alexander Rubinstein, Martin Gubri, Seong Joon Oh arXiv: 2610.08775

Abstract (translated)

Large language models (LLMs) can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. Can LLM agents autonomously create cheaper solutions for such workloads? The authors call this ability "bottling": the ability to turn general capabilities into task-specific solutions that balance answer quality and amortised cost.

The paper introduces BOTTLED, a benchmark in which agents receive an entire unlabelled workload and must complete it under fixed time, compute, and LLM API budgets. Agents choose their own approach, such as training a small model or writing a reusable program.

Key findings

Across ten models and three tasks:

Strong zero-shot performance does not reliably translate into strong bottling capability. Models with similar zero-shot scores can differ widely after bottling.
48 of 60 bottling runs scored below the lower bound of their model's zero-shot performance 95% confidence interval.
Bottling can still deliver major savings: on a query-product relevance classification task, Opus 5 maintained ~82% of its zero-shot macro-F1 at roughly 657x lower reported cost.
Bottling was competitive with Jev, a "System-1" model built specifically for cheap repeated inference.

Conclusion

BOTTLED provides a foundation for evaluating and improving agents' ability to invest limited resources into reusable solutions for large, repetitive workloads.

---

Source: arXiv:2610.08775

*Auto-collected on 2026-10-08*