Gorilla: Large Language Model Connected with Massive APIs
Paper Overview
| Field | Content | |------|------| | Title | Gorilla: Large Language Model Connected with Massive APIs | | Authors | Shishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. Gonzalez (UC Berkeley) | | Published | 2023-05-24 | | Source | https://arxiv.org/abs/2305.15334 | | Project page | https://gorilla.cs.berkeley.edu |
One-Sentence Summary
Gorilla is a finetuned LLaMA-based model that surpasses GPT-4 at writing accurate API calls, and when combined with a document retriever, it adapts to documentation changes while substantially reducing hallucination.
Background and Motivation
While LLMs excel at tasks like mathematical reasoning and program synthesis, their ability to effectively use tools via API calls remains limited. Even state-of-the-art models like GPT-4 struggle due to:
- Inability to generate accurate input arguments;
- Tendency to hallucinate incorrect API usage.
- Gorilla model: A finetuned LLaMA-based model that outperforms GPT-4 on writing API calls.
- Retrieval-aware training: When paired with a document retriever, Gorilla adapts to test-time document changes, supporting user-driven updates and API version changes.
- Hallucination mitigation: Retrieval integration substantially reduces hallucinated or incorrect API usage.
- APIBench benchmark: A comprehensive evaluation dataset covering HuggingFace, TorchHub, and TensorHub APIs.
- Paper: arXiv:2305.15334
- Code, model, data, demo: https://gorilla.cs.berkeley.edu
Gorilla targets this gap, enabling LLMs to call massive collections of APIs reliably.
Original Abstract
> Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call. We release Gorilla, a finetuned LLaMA-based model that surpasses the performance of GPT-4 on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time document changes, enabling flexible user updates or version changes. It also substantially mitigates the issue of hallucination, commonly encountered when prompting LLMs directly. To evaluate the model's ability, we introduce APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and TensorHub APIs. The successful integration of the retrieval system with Gorilla demonstrates the potential for LLMs to use tools more accurately, keep up with frequently updated documentation, and consequently increase the reliability and applicability of their outputs. Gorilla's code, model, data, and demo are available at https://gorilla.cs.berkeley.edu
Key Contributions
Positioning in the Retrieval + LLM Landscape
Gorilla sits at the intersection of information retrieval and LLM tool use. In classic search stacks, recall handles coverage, ranking handles discrimination, and generation handles presentation; the LLM era adds two new variables: inference budget and action space (whether to retrieve, how many times, and which tools to call). Gorilla is an early and influential example of teaching models to use tools reliably, related to later work on retrieval-augmented and agentic search systems such as FreshLLMs, IRCoT, ReSearch, and Search-R1.
Evaluation Context
The evaluation uses APIBench (HuggingFace, TorchHub, TensorHub APIs), measuring exact-match accuracy of generated API calls, with and without retrieved documentation. Quantitative results should be verified against the original PDF tables.
Takeaways
1. Architecture: Retrieval + generation remains the dominant pattern; Gorilla shows retrieval during inference keeps API knowledge fresh. 2. Reliability: Reducing hallucination in tool calls is critical for production LLM systems. 3. Evaluation: Task-level benchmarks like APIBench complement traditional IR metrics. 4. Engineering: Latency, cost, and safety constraints matter when deploying tool-calling LLMs.