Ground Truth.
AI, checked against the source.

Learn · Intermediate

Decision theory: choosing an AI action when mistakes have different costs

Decision theory chooses an action by combining beliefs about possible outcomes with the benefits and costs of each outcome. It matters for AI because a prediction can be accurate on average and still lead to harmful decisions when mistakes have unequal consequences.

A detector that assigns a 90% probability to a fire and a detector that assigns a 90% probability to a spam message produce the same kind of number. The appropriate response can be completely different. Sending an alert, shutting down a machine, seeking a second opinion and doing nothing all carry different costs. A probability alone does not tell an operator which action is best.

From prediction to action

Suppose a classifier estimates the chance that an email is malicious. The operator has two actions: block it or allow it. There are also two possible states: malicious or harmless. Blocking a harmless message creates a false alarm; allowing a malicious one creates a miss. Assign a cost to each. If a miss costs 20 units and a false alarm costs one, then with zero costs for correct decisions, blocking has expected cost 1 − p, while allowing has expected cost 20p, where p is the probability of a malicious message. Blocking is preferred when 1 − p < 20p, or p > 1/21, approximately 4.8%.

That threshold is a mathematical consequence of the assumed costs, not a universal security setting. If blocking can stop an emergency message, the false-alarm cost rises. If a human reviewer can investigate, a third action changes the calculation. The units can represent money, time or another explicitly chosen measure; mixing very different harms into a single number also requires a defensible value judgment.

The same logic helps explain why an alerting system should not be evaluated solely by how often it is right. Precision and recall describe distinct error patterns, and the ROC lesson shows how the operating point changes as a threshold moves. Those statistics are inputs to a decision, not a substitute for deciding who bears the consequences.

Why the base rate matters

A test can look impressive while most alerts are wrong when the event is rare. Imagine 10,000 cases, of which 100 are genuinely dangerous. A test catches 90% of dangerous cases and falsely flags 5% of harmless cases. It catches 90 real cases but also flags 495 of the 9,900 harmless ones. Only 90 of 585 alerts, about 15%, correspond to real danger. The calculation uses the prevalence of danger as well as the test's error rates.

This does not mean the test is useless. If missing a real threat is extremely costly and review is cheap, it can still be worth investigating alerts. It does mean a person seeing a positive result should not assume the probability of danger equals the detector's sensitivity. The Bayesian updating lesson explains how evidence and prior prevalence combine. A model's own probability is useful for these calculations only if it is well calibrated in the environment where it is used. Chuan Guo and colleagues' study of neural-network calibration demonstrates why modern model confidence needs empirical checking.

The decision is a chain

The action after a prediction often involves several people and systems. An alert can be checked, canceled, forwarded or detached from its original context. The complaint in the Kenwood school case alleges that a firearm alert was canceled before an image was later referred to police. The allegations are not adjudicated, but the pleaded sequence illustrates a general design problem: a status update helps only if downstream decision-makers receive and use it.

A good decision model therefore includes the available options and the handoffs, not just a classifier's score. Does a reviewer have enough information to check an alert? Can a cancellation be communicated to everyone who received it? Does a screenshot preserve the current status? If a decision can be reversed only at high cost, extra verification can be valuable even when it adds delay.

Expected cost and its limits

Expected cost averages the costs of outcomes weighted by their probabilities. It is useful for comparing repeatable decisions with explicit assumptions. It can also conceal rare catastrophic outcomes, uncertain cost estimates, unequal effects on different groups, or constraints that should not be traded away for a small average gain. A hospital, for example, may require human review in particular cases regardless of the average score.

Likewise, a reported model confidence is not automatically a trustworthy probability. Calibration can drift when inputs change, labels are incomplete or adversaries adapt. A system that knows it is uncertain may abstain, but abstention needs a reliable escalation path; simply handing every hard case to an overloaded person can move the failure elsewhere.

Decision theory is associated with Abraham Wald's statistical decision framework, including his book Statistical Decision Functions. It provides a disciplined way to state assumptions and compare actions. Its practical value comes from making those assumptions visible: which outcomes are possible, how probabilities were estimated, whose costs count, and what happens after the system acts. A threshold should be reviewed when any of those answers changes.

Key papers
On Calibration of Modern Neural Networks (Guo et al., 2017)

Key questions

How does decision theory choose an AI action?

It compares the expected consequences of available actions using probabilities and explicit costs or benefits.

Why can a low probability still justify an alert?

A low probability can justify an alert when the cost of missing a real danger is much larger than the cost of investigating a false alarm.

Does the best threshold stay fixed?

No. The appropriate threshold can change with prevalence, error costs, available actions and probability calibration.
Cite this

APA

Ground Truth. (2026, November 19). Decision theory: choosing an AI action when mistakes have different costs. Ground Truth. https://groundtruth.day/learn/decision-theory-expected-cost-and-ai-thresholds.html

BibTeX

@misc{groundtruth:decision-theory-expected-cost-and-ai-thresholds,
  title  = {Decision theory: choosing an AI action when mistakes have different costs},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/decision-theory-expected-cost-and-ai-thresholds.html}
}

Topics: decision-theory · evaluation · risk