Cost-Aware Abstention for LLM-Based IT Ticket Classification

The 30-second take

• What: On 47,837 public tickets across eight classes, TF-IDF logistic regression with a calibration-chosen 0.60 threshold cuts test cost from 0.763 to 0.472 under costs (0, 1, 5) and drops wrong routes from 15.3% to 3.5%.
• Why it matters: Cheap auto-routing that does not dump expensive mistakes is still scarce. If a frozen, cost-conditional abstention gate holds, more support queues can turn models on only when expected cost beats a human — a cheaper default, not a certified chatbot.
• Who should care: IT operations leads, applied-ML teams comparing prompted language models with classical classifiers, and anyone who must price a wrong route against a human triage minute.

What the paper actually did

The authors treat cost-aware abstention as an offline, cost-conditional readiness check for IT ticket routing. They select a routing policy on calibration data under explicit costs, freeze that policy, and evaluate it once on held-out test data.

The public corpus has 47,837 tickets and eight classes. They compare four trained non-language-model classifiers with four large language models under zero-shot and few-shot prompting. Under the illustrative cost model (correct, triage, wrong) equal to (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% confidence interval 35.7% to 40.4%), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis.

The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic recalibration lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and a correctness AUROC of 0.500. The authors therefore report correctness AUROC and tie-aware risk-coverage area alongside expected calibration error. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19 of 1,000. An LLM-only agreement control and margin and entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs. The work is an evaluation framework, not a new abstention algorithm or a universal deployment certificate.

What makes this disruptive

If the core claim holds, the scarce capability under pressure is cheap, trustworthy ticket routing that does not quietly send work to the wrong team. The distinctive move is not a new model. It is a frozen, cost-conditional policy chosen on calibration data and scored once on held-out tickets, so “ready to auto-route” is an expected-cost number rather than a demo vibe.

The practical signal is that a classical TF-IDF logistic regression with a 0.60 threshold cuts expected cost by about 38% and wrong routes from 15.3% to 3.5% under costs (0, 1, 5), while the strongest prompted setup is far less accurate than fine-tuned DistilBERT on the same 1,000-ticket slice. Prompted models also collapse at a 0.80 break-even probability: seven defer everything, and Haiku few-shot routes 19 of 1,000.

This preprint ranks 11 of 20 in the 2026-W44 editorial triage (score 72). That ranking is a week-cohort sort for readers of Disruptive Concepts, not peer review and not a verdict on the science.

Why it matters (outside the lab)

Abundance lens (today’s luxuries toward tomorrow’s defaults): expert triage time is scarce, and a wrong automatic route can cost more than that human glance. If a frozen, cost-aware abstention check becomes a default readiness gate, auto-routing is something you turn on only when expected cost beats a person — capable assistance as a software layer, not a scarce staff service.

Near-term, use the preprint to update how teams compare prompted language models with trained classifiers under explicit costs, and to treat calibration metrics as distinct from discrimination. Medium-term, whether this becomes a default operations control depends on other queues, other cost ratios, and independent replication.

No calendar date is implied. Disruptive Concepts writes this explainer free for humans. The interesting question is whether cost-conditional abstention stays a paper method or becomes ordinary ticket-router firmware.

Limitations & open questions

This is a preprint, not a product and not a deployment certificate. The authors themselves call it an evaluation framework, not a new abstention algorithm. Results are scoped to one public eight-class corpus, the listed prompts and model endpoints, and the illustrative costs (0, 1, 5). We have not reproduced the numbers.

The abstract does not claim that prompted language models beat trained classifiers on this task; the opposite accuracy gap is reported on the 1,000-ticket subset. Isotonic recalibration is said not to reliably improve discrimination. Seven prompted configurations defer all tickets at break-even 0.80, which is a limitation of those setups under that cost model, not a universal failure of every language model.

Abundance is not automatic. A cheaper expected-cost number on one corpus does not demonetize IT support on a fixed date. Other ticket taxonomies, drifting categories, and real on-call costs still sit between this framework and a default.