Cost-Aware Abstention for LLM-Based IT Ticket Classification (arXiv:2610.31657)

Forum topic · 小凯 · 2026-10-24

Summary

This paper by Sachin Gupta (arXiv:2610.31657) introduces cost-aware abstention as an offline, cost-conditional readiness assessment framework for IT support ticket routing, where misrouting can cost more than human triage. A routing policy is selected on calibration data under explicit costs and then evaluated once as a frozen policy on held-out test data. On a public corpus of 47,837 tickets across eight classes, the study compares four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under an example cost model (correct, triage, wrong) = (0, 1, 5), TF-IDF with logistic regression and a 0.60 threshold reduced test cost from 0.763 to 0.472 (a 38.1% relative reduction) and cut wrong routes from 15.3% to 3.5%. The frozen policy beat always-route and always-defer baselines in 14 of 16 post-hoc cost settings. The best prompting configuration reached 0.477 accuracy versus 0.879 for fine-tuned DistilBERT. The authors also found isotonic regression lowers expected calibration error without reliably improving discrimination, recommending reporting both correctness AUROC and risk-coverage area.

Paper Overview

Field: NLP Author: Sachin Gupta Published: 2026-10-22 arXiv: 2610.31657

Abstract

Wrong automatic routing of an IT support ticket can cost more than human triage. The paper evaluates cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data.

Key Findings

Setup: On a public corpus of 47,837 tickets across eight classes, four trained non-LLM classifiers are compared with four large language models under zero- and few-shot prompting.
Cost model: Under an example cost model (c_correct, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression at a 0.60 threshold reduces test cost from 0.763 to 0.472 (a 38.1% relative reduction), while lowering the wrong-route rate from 15.3% to 3.5%.
Robustness: The frozen, calibration-selected policy outperforms both "always route" and "always defer" baselines in 14 of 16 post-hoc cost parameter settings.
LLM performance gap: The strongest prompting configuration achieves 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets.
Calibration insight: Isotonic regression reduces expected calibration error but does not reliably improve discrimination; the authors recommend reporting both correctness AUROC and risk-coverage area.

Original Abstract

Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. TF-IDF with logistic regression and a threshold of 0.60 reduces test cost by 38.1%, while lowering wrong routes from 15.3% to 3.5%.