Skip to research
Vraelis Research
All researchResearch methods / Reference note and proposed protocol

Precision and recall under rare failures

Report how alerts behave when the positive class is uncommon.

Research note · November 14, 2026

Research question

Why can a detector with a low false-positive rate still produce many unhelpful alerts?

Method

  1. Define the positive condition and retain the underlying counts of true positives, false positives and false negatives. State the prevalence in the evaluation sample.
  2. Report precision and recall at a threshold chosen independently of the final test set. Include the number of predicted positives so a high precision score cannot hide negligible coverage.
  3. If discussing a different deployment prevalence, label any recalculation as an assumption-based scenario. Do not present a balanced test set’s precision as directly transferable to a rare-event setting.

Records to retain

  • Label definition and prevalence
  • Confusion counts
  • Threshold-selection procedure
  • Alert volume and missed positives

Source basis

Scikit-learn defines precision and recall from different denominators. Precision depends on the mix of positives and negatives encountered, so it must be interpreted with the evaluation population and alert volume.

Interpretation & limits

No detector or measured curve is introduced by this note. File-digest comparisons are deterministic checks and should not be dressed up as scored classifiers without a relevant statistical task.

Next experiments

  • Publish confusion counts alongside any future aggregate metric.
  • Test whether the chosen threshold supports a useful review workload.

References

  1. Scikit-learn model evaluation
  2. Scikit-learn cross-validation

Technical reference

Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.

Related notes