Define the evaluation population
A result needs a stated population before its score can be interpreted.
Research note · November 14, 2026Research question
Which systems and conditions does an evaluation actually represent?
Method
- Define the unit of evaluation, such as a file, release, device, session or scenario. List inclusion and exclusion rules before selecting cases.
- Record how the sample was obtained and which conditions are absent. Keep convenient examples distinct from a representative sample of a deployment population.
- Report results by relevant strata as well as in aggregate. A large number of near-identical cases should not hide the absence of an important environment or failure mode.
Records to retain
- Sampling unit and eligibility rules
- Selection procedure
- Population and sample differences
- Case-level outcomes
Source basis
NIST’s AI RMF emphasizes context and intended use in risk management. Scikit-learn’s cross-validation guidance similarly shows that evaluation structure must reflect dependencies in the data. Neither makes a convenience sample representative by itself.
Interpretation & limits
This protocol does not supply a dataset or an estimate of real-world failure frequency. Claims should remain bounded by the cases and conditions actually evaluated.
Next experiments
- Write the population definition before collecting more fixtures.
- Identify one missing stratum that could change the conclusion.
References
Technical reference
Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.