Most clinical AI benchmarks reward being right. A new proposal argues they should also reward declining to answer, and offers a scoring scheme that penalizes confident responses on cases outside a model's validated scope.

The motivation is drawn from deployment reports rather than theory. In audits of triage and decision-support tools, the errors that reached patients were rarely cases where the model was uncertain and wrong. They were cases where the model was confident and the case did not belong to it — a pediatric presentation routed to an adult model, a post-transplant patient scored by a general risk tool.

The proposed benchmark constructs test sets that deliberately include out-of-distribution cases and measures the rate at which models correctly hand off rather than answer. Early results across a dozen open models are poor. Abstention rates on clearly out-of-scope cases ranged from four to thirty-one percent.

The critique already surfacing is that abstention is easy to game — a model that abstains constantly scores well on the safety axis and is useless. The authors acknowledge this and pair the metric with a coverage requirement, which reviewers describe as necessary but not obviously sufficient.