Gorilla: A Finetuned LLaMA Model That Surpasses GPT-4 at Writing API Calls

Forum topic · 小凯 · 2026-07-05

Summary

Gorilla is a large language model finetuned from LLaMA by researchers at UC Berkeley (Patil, Zhang, Wang, Gonzalez; arXiv:2305.15334, May 2023) that surpasses GPT-4 at generating accurate API calls. LLMs often hallucinate incorrect tool usage or produce wrong input arguments; Gorilla addresses this via retrieval-aware training. Combined with a document retriever, the model adapts to test-time documentation changes, supporting user updates and API version changes while substantially mitigating hallucination. The authors introduce APIBench, a benchmark spanning HuggingFace, TorchHub, and TensorHub APIs to evaluate tool-calling ability. The work demonstrates that integrating retrieval with LLMs enables more reliable tool use, keeps outputs consistent with frequently updated documentation, and increases the applicability of LLM-generated API calls in real systems. Code, model weights, data, and a demo are available at https://gorilla.cs.berkeley.edu. This Chinese forum post summarizes the paper's motivation, architecture, evaluation design, and implications for retrieval-augmented and agentic LLM systems.

Gorilla: Large Language Model Connected with Massive APIs

Paper: Gorilla: Large Language Model Connected with Massive APIs Authors: Shishir G. Patil, Tianjun Zhang, Xin Wang, Joseph E. Gonzalez (UC Berkeley) Published: 2023-05-24 Link: https://arxiv.org/abs/2305.15334 Project page: https://gorilla.cs.berkeley.edu

One-line summary

A finetuned LLaMA-based model that outperforms GPT-4 at writing API calls, with retrieval integration to handle documentation updates and reduce hallucination.

Background and Motivation

LLMs have advanced rapidly on tasks like mathematical reasoning and program synthesis, but their ability to effectively use tools via API calls remains limited. Even state-of-the-art models like GPT-4 struggle because they cannot reliably generate accurate input arguments and tend to hallucinate incorrect API usage. Gorilla was proposed to close this gap, situating itself at the intersection of information retrieval and LLM tool use.

Core Contributions

Gorilla: a finetuned LLaMA-based model that surpasses GPT-4 on writing API calls.
Retrieval-aware adaptation: when combined with a document retriever, Gorilla adapts to test-time document changes, enabling flexible user updates and API version changes.
Hallucination mitigation: substantially reduces the incorrect API usage common when prompting LLMs directly.
APIBench: a comprehensive benchmark dataset covering HuggingFace, TorchHub, and TensorHub (TensorHub as stated in the original abstract) APIs.

Method Overview

The approach follows a problem formulation → model design → training → inference pipeline:

1. Input representation: encode queries and API documentation into structured prompts. 2. Core modules: retriever and generator (finetuned LLaMA) working together. 3. Training: finetuning on API-call data. 4. Inference: optionally retrieve up-to-date API documentation before generating the call, keeping outputs aligned with current docs.

Original abstract (English)

> Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call. We release Gorilla, a finetuned LLaMA-based model that surpasses the performance of GPT-4 on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time document changes, enabling flexible user updates or version changes. It also substantially mitigates the issue of hallucination, commonly encountered when prompting LLMs directly. To evaluate the model's ability, we introduce APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and TensorHub APIs. The successful integration of the retrieval system with Gorilla demonstrates the potential for LLMs to use tools more accurately, keep up with frequently updated documentation, and consequently increase the reliability and applicability of their outputs. Gorilla's code, model, data, and demo are available at https://gorilla.cs.berkeley.edu

Evaluation

Specific numbers should be verified against the original PDF tables. The evaluation design centers on APIBench and comparisons against zero-shot LLM baselines with and without retrieval.

Key Takeaways

1. Architecture: retrieval + generation cascades remain mainstream; agentic paradigms are making retrieval strategy itself learnable. 2. Data: high-quality instruction data matters as much as logs; synthetic data requires leakage and distribution-shift safeguards. 3. Evaluation: offline metrics diverge from online satisfaction; LLM-as-judge needs cross-validation with human evaluation. 4. Deployment: latency, cost, interpretability, and safety are hard constraints beyond benchmark optimization.

Limitations and Future Work

Potential limitations include experiment scale constrained by GPU budgets, benchmarks that may not match real user distributions, and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs, and causal/fairness constraints for recommender applications.