Skip to research
Vraelis Research
All researchResearch methods / Reference note and proposed protocol

Confidence scores and calibration

Check whether predicted probabilities correspond to observed frequencies.

Research note · November 14, 2026

Research question

Does a score of 0.8 mean an event occurs about eight times in ten under the evaluated conditions?

Method

  1. Collect probabilities and outcomes on data not used to fit the model or its calibration mapping. Preserve the evaluation population and class prevalence.
  2. Inspect a reliability diagram with sample counts per bin. Report the binning policy and avoid interpreting sparsely populated bins as precise evidence.
  3. Compare calibration and discrimination separately. A model can rank examples usefully while assigning misleading probabilities, or appear calibrated while offering little useful separation.

Records to retain

  • Predicted probabilities and labels
  • Calibration fitting split
  • Reliability-bin definitions and counts
  • Discrimination and calibration measures

Source basis

Scikit-learn’s calibration documentation distinguishes calibrated probabilities from arbitrary confidence scores and describes reliability diagrams and calibration methods. The quality of a ranking and the meaning of its numeric probabilities are different questions.

Interpretation & limits

Calibration measured on one population may not hold after distribution shift. This protocol does not establish that any Vraelis feature emits calibrated risk probabilities.

Next experiments

  • Test calibration on a held-out group or later period.
  • Report sparse regions where probability interpretation remains uncertain.

References

  1. Scikit-learn probability calibration
  2. Scikit-learn cross-validation

Technical reference

Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.

Related notes