Confidence scores and calibration
Check whether predicted probabilities correspond to observed frequencies.
Research note · November 14, 2026Research question
Does a score of 0.8 mean an event occurs about eight times in ten under the evaluated conditions?
Method
- Collect probabilities and outcomes on data not used to fit the model or its calibration mapping. Preserve the evaluation population and class prevalence.
- Inspect a reliability diagram with sample counts per bin. Report the binning policy and avoid interpreting sparsely populated bins as precise evidence.
- Compare calibration and discrimination separately. A model can rank examples usefully while assigning misleading probabilities, or appear calibrated while offering little useful separation.
Records to retain
- Predicted probabilities and labels
- Calibration fitting split
- Reliability-bin definitions and counts
- Discrimination and calibration measures
Source basis
Scikit-learn’s calibration documentation distinguishes calibrated probabilities from arbitrary confidence scores and describes reliability diagrams and calibration methods. The quality of a ranking and the meaning of its numeric probabilities are different questions.
Interpretation & limits
Calibration measured on one population may not hold after distribution shift. This protocol does not establish that any Vraelis feature emits calibrated risk probabilities.
Next experiments
- Test calibration on a held-out group or later period.
- Report sparse regions where probability interpretation remains uncertain.
References
Technical reference
Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.