Symptom checkers are typically evaluated by comparing their disposition recommendation against a clinician panel. A study tracking what patients did after using one suggests that this measures something other than the tool's effect.

Among 2,900 users who received a self-care recommendation for symptoms the panel agreed were appropriate for self-care, thirty-eight percent sought in-person care within seventy-two hours anyway. Among those who received an urgent recommendation, twelve percent did not seek care at all.

The free-text follow-up is the most useful part of the study. Patients who overrode a self-care recommendation overwhelmingly cited not the recommendation itself but the absence of a reason they found convincing. Patients who ignored an urgent recommendation cited cost, childcare and work.

The authors' conclusion is that triage accuracy is necessary and nowhere near sufficient, and that a tool which cannot address the actual determinants of care-seeking should not be evaluated as though disposition advice were its output. Several vendors have begun adding cost estimation and appointment booking directly into the triage flow, which the authors describe as the right instinct without yet having the evidence to support it.