Confidence Audit of Typed Decision Models: Promises vs. Reality
ORIGINAL / Confidence Is Not a Promise: A Reference Audit of Typed Decision Models
This study audits the reliability of confidence promises in typed decision models through large-scale experiments, revealing that point-estimate threshold selection often breaks promises while methods like Learn then Test keep them, and analyzes multilingual calibration, noise robustness, and drift scenarios.
01 ABSTRACT
The paper measures what probability distributions returned by typed decision models license for decisions, finding that point-estimate thresholds break promises in 147 of 190 evaluations while Learn then Test keeps them in all. Calibrated API models keep a 95% promise in only 8 of 51 locales, and ordinary typing noise breaks promises calibrated on clean text. The authors argue that confidence licenses statements about groups of decisions, not guarantees for individual decisions.
02 KEY FINDINGS
- Point-estimate thresholds break promises in 147/190 evaluations; Learn then Test keeps them in all.
- Jev model keeps a 95% promise in only 8 of 51 locales; typing noise breaks calibrated promises.
- Fine-tuned Laya needs 250 to 8,000 labels to be non-inferior to Jev in four domains, and does not reach it on LIAR2.
AI GENERATED SUMMARY / DISCOVERED BY ARXIV CS.LG