What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation in Tool Calling

Forum topic · 小凯 · 2026-09-17

Summary

A new arXiv paper (2609.17515) systematically studies how pruning degrades large language model performance in smart-home tool calling. The authors evaluate four LLMs spanning dense Transformer, dense hybrid, and mixture-of-experts (MoE) architectures, applying depth, width, hybrid, and expert pruning methods followed by post-pruning supervised fine-tuning. They assess over 19,500 instances from three smart-home datasets, characterizing degradation along two dimensions: action components (operation, device, argument, and value) and task complexity. Key findings: dense models exhibit narrow safe pruning regions followed by sharp degradation, whereas MoE models tolerate substantially more pruning. Pruning first damages specificity (such as argument and value precision) before eroding pattern-level intent recognition, and aggressive dense pruning can trigger systematic over-rejection of valid commands. The results argue that overall task accuracy is insufficient for selecting pruned LLMs for reliable tool execution, and that finer-grained evaluation across action components and complexity levels is needed.

Overview

Field: NLP Authors: Congjing Zhang, Vashishtha Patil, Henning Lange, Usman Aleem Published: 2026-09-15 arXiv: 2609.17515

Summary

Pruning can reduce the deployment cost of large language models (LLMs), but its impact on context-grounded tool calling remains poorly understood. This paper systematically studies pruning-induced degradation in smart-home tool calling across four LLMs spanning dense Transformer, dense hybrid, and mixture-of-experts (MoE) architectures, together with depth, width, hybrid, and expert pruning methods.

Methodology

After post-pruning supervised fine-tuning (SFT), the authors evaluate more than 19,500 instances from three smart-home datasets.
Beyond aggregate task accuracy, degradation is characterized along two dimensions:
Action components: operation, device, argument, and value
Task complexity

Key findings

Dense models have narrow safe pruning regions followed by sharp degradation, while MoE models tolerate considerably more pruning.
Pruning first breaks specificity (fine-grained details such as arguments and values) before harming pattern-level intent recognition.
Aggressive dense pruning may even induce systematic over-rejection of valid commands.

Conclusion

These findings highlight the importance of evaluating beyond aggregate accuracy when selecting pruned LLMs for reliable tool execution.

--- *Auto-collected on 2026-09-17*

What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation in Tool Calling

Summary

Overview

Summary

Methodology

Key findings

Conclusion

Tags