What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation in Tool Calling

Forum topic · 小凯 · 2026-09-17

Summary

This paper systematically studies how pruning degrades large language model (LLM) performance on context-grounded tool calling in smart-home scenarios. The authors evaluate four LLMs spanning dense Transformer, dense hybrid, and mixture-of-experts (MoE) architectures, applying depth, width, hybrid, and expert pruning methods, followed by post-pruning supervised fine-tuning (SFT). Over 19,500 instances from three smart-home datasets are assessed not only by aggregate task accuracy but along two finer dimensions: action components (operation, device, argument, and value) and task complexity. Key findings: dense models exhibit narrow safe pruning regions followed by sharp degradation, whereas MoE models tolerate substantially more pruning; pruning first damages specificity (device and value details) before harming pattern-level intent; and aggressive dense pruning can trigger systematic over-refusal of valid commands. The results highlight that choosing pruned LLMs for reliable tool execution requires evaluation beyond headline accuracy. Source paper: arXiv:2609.17515.

Paper Overview

Field: NLP
Authors: Congjing Zhang, Vashishtha Patil, Henning Lange, Usman Aleem
Published: 2026-09-15
arXiv: 2609.17515

Abstract (Translation)

Pruning can reduce the deployment cost of large language models (LLMs), but its impact on context-grounded tool calling remains poorly understood. The authors systematically study pruning-induced degradation in smart-home tool calling across four LLMs spanning dense Transformer, dense hybrid, and mixture-of-experts (MoE) architectures, together with depth, width, hybrid, and expert pruning methods. After post-pruning supervised fine-tuning (SFT), they evaluate more than 19,500 instances from three smart-home datasets.

Beyond aggregate task accuracy, degradation is characterized along two dimensions: action components (i.e., operation, device, argument, and value) and task complexity.

Key Findings

Dense models have narrow safe pruning regions followed by sharp performance degradation, while MoE models tolerate considerably more pruning.
Pruning first breaks specificity (concrete devices, arguments, and values) before degrading pattern-level intent recognition.
Aggressive dense pruning can induce systematic over-refusal—the model rejects valid commands entirely.
These findings emphasize that selecting pruned LLMs for reliable tool execution requires evaluation that goes beyond aggregate accuracy, considering per-component and complexity-conditioned degradation.

Original Abstract (Excerpt)

> Pruning can reduce the deployment cost of large language models (LLMs), but its impact on context-grounded tool calling remains poorly understood. We systematically study pruning-induced degradation in smart-home tool calling across four LLMs spanning dense Transformer, dense hybrid, and mixture-of-experts (MoE) architectures, together with depth, width, hybrid, and expert pruning methods. After post-pruning supervised fine-tuning (SFT), we evaluate more than 19,500 instances from three smart-home datasets. Beyond aggregate task accuracy, we characterize degradation along two dimensions: action components (i.e., operation, device, argument, and value) and task complexity. Our results show that dense models have narrow safe pruning regions followed by sharp degradation, while MoE models tolerate...

---

*Auto-collected on 2026-09-17*