Skip to research
Vraelis Research
All researchResearch methods / Reference note and proposed protocol

Keep related cases together

Avoid measuring familiarity with a device or session as if it were generalization.

Research note · November 14, 2026

Research question

What happens when related records are split across training and evaluation?

Method

  1. Identify grouping variables such as device, source folder, recording session or customer environment. Decide which groups must remain entirely outside training and threshold selection.
  2. Compare a random row split with a group-aware split using the same underlying records. Preserve the group assignments and report the difference without selecting whichever score looks better.
  3. Check for derived copies and near-duplicates across the boundary. Transformations of the same source can leak information even when their filenames differ.

Records to retain

  • Group identifiers
  • Train and evaluation assignments
  • Duplicate-detection procedure
  • Results under each split

Source basis

Scikit-learn documents grouped cross-validation for samples with shared structure. A row-wise split can place dependent examples on both sides, producing an overly optimistic view of performance on new groups.

Interpretation & limits

Grouping is not a complete defense against leakage. Shared preprocessing, label construction or source overlap can still contaminate the evaluation. No comparative score is reported here.

Next experiments

  • Choose groups that match the intended deployment boundary.
  • Audit transformations that create multiple records from one source.

References

  1. Scikit-learn cross-validation
  2. Scikit-learn common pitfalls

Technical reference

Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.

Related notes