calibration
A reinforcement-learning critic can supply confidence, under specific conditions News
A new study reuses a reinforcement-learning value critic to estimate answer correctness, with gains that depend on the reward and training setup.
A small teaching study finds confident AI falsehoods survive repeated questioning News
A 32-person experiment found that learners often absorbed planted AI falsehoods while remaining confident, under instructions to rely only on the assistant.
Microsoft launches Decision-1 to score workflow choices without long generated answers News
Microsoft-Decision-1 returns probabilities for bounded workflow decisions through hosted services, while H2O offers a separate downloadable decision model.
Selective prediction and abstention: when an AI should decline to decide Lesson
Selective prediction lets a model answer only when its expected reliability clears a threshold, routing uncertain cases to a person, a stronger model, or another check instead of forcing every input into an answer.
Jeff releases a small open model for fast typed decisions on local hardware News
Jeff is an open 0.8B decision-model project that returns calibrated option choices instead of prose, illustrating where a small local model can be useful without claiming frontier reasoning ability.
A 25-line Jev parody shows why closed choices are not calibrated decisions News
A local-Qwen demonstration can reproduce Jev's visible closed-choice probability interface, but it does not evaluate the specialized model's claimed decision quality, calibration, or parallel serving.
Kev brings Jev-like local decision models to Qwen, while audits challenge universal calibration claims News
Kev released open local models for typed probability decisions, as published evaluations and an out-of-distribution audit show that confidence calibration varies sharply by task.
TypeSafe launches Jev, a hosted model for bounded software decisions News
TypeSafe’s early-access Jev turns predefined choices, scores and probabilities into fast API calls, but its proprietary architecture and reliability claims remain unverified outside company-run evaluations.
Label smoothing: teaching a classifier not to be certain beyond the evidence Lesson
Label smoothing replaces a perfectly one-hot training target with a slightly softened probability distribution, discouraging a classifier from treating every labeled example as proof that all alternatives are impossible.
Calibration: whether a model's confidence means anything Lesson
A model is calibrated when the things it says with 70% confidence turn out to be true about 70% of the time - and calibration is a completely separate property from accuracy, which is why a model can be both frequently right and systematically untrustworthy about when it is wrong.
Kev Tool
Apache-2.0 local Qwen-based models that return typed probability decisions for yes/no, choice, and rating tasks.
Jeff Tool
Open code and downloadable small models for structured option selection, probabilities, and low-latency local decision loops.
H2O-Lightning-4B Tool
Apache-2.0 decision checkpoint with 9.1 GB of weights; the card reports local measurements on a 32 GB professional GPU, without establishing a minimum.