How AI Turns Probabilities Into Decisions
How costs, probability thresholds, and the option to ask for human review shape AI decisions, and how to evaluate the consequences. This audio was created with Boson AI's Higgs Audio — https://www.boson.ai/higgs-audio
Listen on Spotify · Watch on YouTube · Pocket Casts
From an alarm score to a decision
Eris: <|prosody:expressive_high|>An AI smoke alarm can have good reason to sound even when its model estimates that a fire is unlikely. In this imagined example, a missed fire costs much more than a false alarm, so the decision depends on both the estimated chance and the consequences.
Vestra: Imagine a sensor watching for smoke. The true state is whether there really is a dangerous fire or there isn't. The available actions in this simplified example are to sound an alarm or stay quiet. An outcome combines the action with the state: an alarm during a fire, an unnecessary alarm, a missed fire, or a quiet system when there is no fire.
Eris: A prediction is the estimate before we act. Suppose the model looks at sensor readings and gives a probability score: its estimated chance that a fire is present. A score of twenty percent means the model estimates a one-in-five chance, not that it has seen the true state. The decision is what the system does with that estimate. <|emotion:surprise|>That feels counterintuitive: a one-in-five chance, and we might still sound the alarm.
Vestra: <|prosody:expressive_high|>The missed fire is what worries me in that choice. To make the choice explicit, write a cost table: a cost for every combination of action and true state. Cost can stand for disruption, delay, injury or other consequences expressed on a common scale. <|prosody:pause|>For illustration only, let's give a false alarm a cost of one, a missed fire a cost of nine, and both correct actions a cost of zero. These are invented units, not measured fire-safety costs.
Eris: With the model's one-in-five fire estimate, sounding the alarm has an expected cost of four-fifths of a unit: there's a four-in-five chance of an unnecessary alarm costing one. Staying quiet has an expected cost of one and four-fifths: a one-in-five chance of a missed fire costing nine. <|emotion:surprise|>So the lower expected cost is to sound the alarm, even though no fire is the more likely state. What does that average tell us about this one alarm?
Vestra: Expected cost means you take each possible consequence of an action, weight it by the estimated chance of that state, and add the results. Then compare that total across actions. <|prosody:pause|>It's an average over possibilities, not a promise about the single case in front of you. Its recommendation depends on the estimates and the table being appropriate.
Eris: <|prosody:expressive_high|>That distinction changes what a score means operationally. The same estimated chance can lead to a different action when the consequences change. The model supplies an estimate; a decision policy is the rule that turns it into an action.
Why the cutoff moves
Eris: A threshold is a cutoff in a model's score. In our simple alarm example, the policy sounds the alarm when the estimated fire probability crosses the cutoff, and stays quiet below it. With a false-alarm cost of one and a missed-fire cost of nine, that dividing line is <|prosody:pause|>one in ten.
Vestra: At one in ten, both actions have the same expected cost: nine-tenths of a unit. Above it, the expected cost of staying quiet is larger. Below it, the expected cost of an alarm is larger. The exact handling of a tie can be chosen by the implementation.
Eris: <|emotion:surprise|>That cutoff is much lower than fifty-fifty because we've made a missed fire nine times as costly as a false alarm. <|prosody:expressive_high|>If we reverse those two costs, the cutoff rises to nine in ten. The same score can now lead to silence. I find that reversal striking; the estimate hasn't changed.
Vestra: <|prosody:expressive_high|>But we need to keep the assumptions attached. That simple formula assumes only two states, two actions, fixed error costs and zero cost for correct choices. Different costs for correct actions, or more available actions, can change the rule. A halfway cutoff is appropriate under particular assumptions, including equal costs for the two errors, but it isn't a universal operating rule.
Eris: Let's name those errors. A false positive is an alarm when there is no fire. A false negative is silence when there is a fire. The other two combinations are a true positive, an alarm during a fire, and a true negative, silence when there is none. A confusion matrix is simply a table counting those four outcomes against what actually happened.
Vestra: Raise the alarm threshold and, for the same set of scores, fewer cases trigger alarms. That usually means fewer false alarms but also fewer fires caught, so more fires may be missed. Lower it and you usually catch more fires while raising more false alarms. The exact counts depend on how the scores are distributed.
Eris: <|prosody:expressive_high|>The unnecessary alarms worry me here. Frequency matters too. If fires are rare, there are many more opportunities for false alarms than missed fires. Even a small false-alarm rate can produce a large number of unnecessary alarms. The base rate is how common the event is in the population being considered; it helps determine the probabilities and the actual balance of outcomes. Does that mean the cutoff needs another adjustment for rarity?
Vestra: <|sfx:humming|>Hmm, if the score already is the correct probability for this case in the current population, the cost-based cutoff itself doesn't need an extra base-rate adjustment. <|prosody:pause|>But a model trained or tested where hazards occur at a different frequency may give unsuitable probabilities in the new setting. And a cutoff chosen by counting errors on an unrepresentative sample can give a misleading picture of the real workload.
More than two possible labels
Eris: The same expected-cost rule works when the model has several possible labels. Each label is a category the model can assign, and each possible true category gets its own estimated probability. <|prosody:expressive_high|>Instead of automatically picking the most likely label, we can compare the expected cost of each available action.
Vestra: Consider a made-up three-category inspection task. The model estimates A at forty percent, B at thirty-five and C at twenty-five. Suppose choosing A costs ten units if the truth is B, one if it's C, and zero if it's A. Its expected cost is <|prosody:pause|>three and three-quarters. Choosing B costs one unit if the truth is A or C and zero if it's B, for an expected cost of sixty-five hundredths. <|prosody:expressive_high|>On this invented table, B is the less costly choice even though A is the most likely label.
Eris: <|emotion:surprise|>B beats the most likely label on cost. Can we call B the winner?
Vestra: A third option would also need to be checked before declaring the overall winner.
Eris: <|prosody:expressive_high|>And the table can be directional. Confusing A for B can carry a different consequence from confusing B for A. Chen and colleagues' paper on cost-sensitive training uses medication images to illustrate that point: mixing up two products in one direction can have a different potential harm from mixing them up in the other.
Vestra: Their actual experiment classified photographs of pills inside medication bottles. A pharmacist identified several particularly consequential confusions and assigned them higher costs. Compared with their ordinary image classifier and two cost-sensitive alternatives, the authors report that their method reduced total cost on their test images while keeping overall accuracy similar.
Eris: Similar accuracy sounds reassuring. But did the critical mistakes disappear?
Vestra: <|prosody:pause|>Some critical errors remained. This was an image classification experiment with a particular dataset and expert-assigned costs, not a demonstration of a complete dispensing system.
Eris: <|prosody:expressive_high|>That's the part that worries me: the critical errors that remain. The important separation still holds: a model may output scores for every category, while a policy decides whether to use the highest score, calculate cost for each label, or do something else entirely. The table and the policy are choices made for the task.
When the system declines to decide
Eris: There can be a third action in our alarm example: ask a person to review the sensor reading, or collect more information before deciding. A system that declines to make its usual prediction is abstaining, also called rejecting a case. That needn't mean doing nothing; what happens next has to be designed. What does handing the case to a person cost?
Vestra: <|prosody:expressive_high|>Review has a cost too: staff time, delay, and the possibility that the reviewer makes a mistake. If we can express the likely consequences of review, we can compare its expected cost against sounding the alarm and staying quiet. The cheapest action may be review for some cases, but the answer depends on <|prosody:pause|>what the review actually achieves.
Eris: Another approach starts from a desired error level rather than assigning a numerical price to review. Geifman and El-Yaniv studied selective classification: a classifier makes predictions on some cases and abstains on others. Coverage is the fraction it handles automatically. Selective error is the fraction of its accepted predictions that are wrong.
Vestra: They took trained image classifiers and ranked examples by a confidence score, then selected a cutoff using a separate labeled sample. They compared using the highest output score with a method that repeatedly perturbs the network and measures variation. <|emotion:enthusiasm|>On their image tests the simple highest-score ranking was similar on two datasets and better on their larger image task. <|prosody:expressive_high|>Rejecting lower-ranked cases reduced error among accepted cases, at the expense of handling fewer images automatically. What does their statistical guarantee actually cover?
Eris: Their procedure offers a statistical bound on the error among accepted cases under its sampling assumptions, with a specified chance that the bound fails. <|prosody:pause|>It isn't a promise that every accepted prediction is correct, or that a bound transfers to a changed population. And their experiments counted errors equally; they didn't separately guarantee a low false-alarm rate or a low missed-hazard rate.
Vestra: <|prosody:expressive_high|>That's why I want both numbers in view. Coverage and error have to be read together. A system can look very accurate on accepted cases by sending almost everything elsewhere. It can also accept almost everything and quietly pass risky cases through. Whether the tradeoff helps depends on the actual review process as well as the model.
Teaching a model which errors matter
Eris: You can change a decision policy after training without changing the model. Scikit-learn's documentation describes testing different cutoffs against a chosen measure of consequences, then choosing one that performs well. <|prosody:expressive_high|>The underlying scores stay the same, but the actions can change.
Vestra: <|emotion:determination|>Don't choose that cutoff on the same examples used to fit the model. It may look good because it has adapted to quirks in those examples, a problem called overfitting. Use separate validation data, or cross-validation: repeatedly train on one portion and choose using held-out portions. Then assess the chosen system on an untouched test set representative of where it will be used.
Eris: Cost-sensitive learning can also change the training itself. Training adjusts a model to reduce a loss, a numerical penalty for its predictions. Giving different errors different weights tells the training process which mistakes matter most, though it doesn't guarantee that the trained model will avoid them. <|sfx:sigh|>Uh, I wish assigning a bigger penalty were enough.
Vestra: Charoenphakdee and colleagues studied an explicit rejection cost. In their simplified setup every wrong label has the same cost, a correct label has no cost, and rejection has a fixed smaller cost. With reliable probabilities, you'd compare the cost of rejecting with the chance and cost of a wrong label. <|prosody:expressive_high|>Their method instead trains a set of cost-sensitive classifiers, each distinguishing one class from all the others, so it can learn when to accept or reject without first estimating every class probability.
Eris: <|prosody:expressive_high|>What happens if those separate classifiers don't point to one clear answer?
Vestra: Their rule rejects when none selects a class, or when more than one does. The authors tested it against other rejection methods on text, tabular data and images, including cases with deliberately corrupted training labels. <|prosody:pause|>Their results varied by task and training loss; in several noisy-label settings one version had lower combined error-and-rejection cost. They also observed a comparison method that rejected too few ambiguous cases on some multiclass tasks. In practice, a review workflow may need case-dependent costs and measurements of what happens after referral; the paper's fixed rejection cost doesn't capture those details.
Eris: Chen and colleagues approached cost-sensitive training from another direction. They found that simply penalizing selected mistakes didn't improve those errors in their experiments when a large image model had already fit its training examples almost perfectly. <|emotion:surprise|>If it already gets the examples right, a penalty on those same examples may give little help with errors on new cases. What can the extra training work on, then?
Vestra: <|emotion:enthusiasm|>Their method creates small targeted changes to training images that push the model toward a particular costly wrong label. Those altered examples are then used in further training, with greater weight for more costly confusions. The goal is to move the model's decision boundaries, the divisions between its predicted labels, away from expensive errors.
Eris: <|prosody:expressive_high|>I like how specifically that targets the costly mistake. Does pushing harder keep helping?
Vestra: The researchers report lower cost in their tested image tasks, but the strength of that extra training matters: <|prosody:pause|>too much emphasis on altered examples increased cost in one experiment. It is a method to test for a particular problem, not a universal fix.
Evaluating the whole decision
Eris: Aggregate accuracy, the fraction of all labels that are correct, can hide costly failures. <|prosody:expressive_high|>If almost every case has no hazard, a system that always says no hazard can look accurate while missing every real one. And an image model can have similar overall accuracy to another while making very different medication confusions. That's what makes a reassuring accuracy figure so uncomfortable here. What would you want to see alongside it?
Vestra: A useful evaluation starts with the stated cost table and measures the consequences of the complete policy on cases with known outcomes. Count false positives and false negatives separately, and for several categories inspect which true label was mistaken for which other label. Measure coverage, the fraction handled automatically, and the fraction sent for review. <|prosody:pause|>For accepted cases, measure errors and their costs; for referred cases, account for the review's delay and outcomes too.
Eris: <|prosody:expressive_high|>Even that isn't enough for me. The table itself deserves scrutiny. A single cost can leave out harms to people affected by a false alarm, or treat a delayed review as free. Different people can bear different consequences. A mathematically lowest expected cost is only as useful as the consequences included in the calculation.
Vestra: <|emotion:determination|>And I'd want to know where those results hold. Test under the conditions of use: the kinds of cases, event frequencies, sensors and review process the system will actually encounter. Probability estimates can be wrong, and an apparently optimal cutoff can stop being appropriate when costs or the mix of cases change. Check performance again when conditions change, and revisit the policy or model using fresh evidence.
Eris: <|prosody:expressive_high|>The practical question is not just which answer the model thinks is most likely. It's what happens after each possible choice, including the choice to ask for help, and whether the evidence supports using that rule in the setting where people will live with the consequences.
Sources and follow
Eris: <|prosody:expressive_high|>For today's other AI stories, listen to the separate news brief. You'll find the sources and daily coverage at Ground Truth, groundtruth.day. Follow AI in Detail, and tell us: <|prosody:pause|>where would you want an AI system to ask for human review?
Vestra: <|emotion:contentment|>Thanks for listening.