Precision and recall under rare failures
Report how alerts behave when the positive class is uncommon.
Research note · November 14, 2026Research question
Why can a detector with a low false-positive rate still produce many unhelpful alerts?
Method
- Define the positive condition and retain the underlying counts of true positives, false positives and false negatives. State the prevalence in the evaluation sample.
- Report precision and recall at a threshold chosen independently of the final test set. Include the number of predicted positives so a high precision score cannot hide negligible coverage.
- If discussing a different deployment prevalence, label any recalculation as an assumption-based scenario. Do not present a balanced test set’s precision as directly transferable to a rare-event setting.
Records to retain
- Label definition and prevalence
- Confusion counts
- Threshold-selection procedure
- Alert volume and missed positives
Source basis
Scikit-learn defines precision and recall from different denominators. Precision depends on the mix of positives and negatives encountered, so it must be interpreted with the evaluation population and alert volume.
Interpretation & limits
No detector or measured curve is introduced by this note. File-digest comparisons are deterministic checks and should not be dressed up as scored classifiers without a relevant statistical task.
Next experiments
- Publish confusion counts alongside any future aggregate metric.
- Test whether the chosen threshold supports a useful review workload.
References
Technical reference
Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.