Keep related cases together
Avoid measuring familiarity with a device or session as if it were generalization.
Research note · November 14, 2026Research question
What happens when related records are split across training and evaluation?
Method
- Identify grouping variables such as device, source folder, recording session or customer environment. Decide which groups must remain entirely outside training and threshold selection.
- Compare a random row split with a group-aware split using the same underlying records. Preserve the group assignments and report the difference without selecting whichever score looks better.
- Check for derived copies and near-duplicates across the boundary. Transformations of the same source can leak information even when their filenames differ.
Records to retain
- Group identifiers
- Train and evaluation assignments
- Duplicate-detection procedure
- Results under each split
Source basis
Scikit-learn documents grouped cross-validation for samples with shared structure. A row-wise split can place dependent examples on both sides, producing an overly optimistic view of performance on new groups.
Interpretation & limits
Grouping is not a complete defense against leakage. Shared preprocessing, label construction or source overlap can still contaminate the evaluation. No comparative score is reported here.
Next experiments
- Choose groups that match the intended deployment boundary.
- Audit transformations that create multiple records from one source.
References
Technical reference
Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.