Testing a specific claim

Research question

What must be recorded for someone else to assess a result and understand where it stops applying?

Method

1. State one claim and the system boundary it concerns. Separate file integrity, release authorization, runtime reporting and model behavior; each needs different evidence.
2. Define the baseline, normal variation and fault conditions. Record attacker access where relevant, along with what the test does not simulate.
3. Freeze the data split, fixture revision, software environment and expected outcomes before running the evaluation. Keep failures and inconclusive observations in the record.
4. Choose metrics that answer the claim. Deterministic integrity checks need case-level outcomes; a scored detector would additionally need labeled positives and negatives, threshold selection and false-positive/false-negative analysis.

Records to retain

• Claim, threat model and test boundary
• Data origin, fixture revision and selection criteria
• Environment and reproducible procedure
• Raw outcomes, exclusions and uncertainty

Current evidence

This is a proposed reporting protocol grounded in the published evaluation guidance. It contains no newly measured detection rates, ROC curves or performance results.

Interpretation & limits

An aggregate score can hide untested conditions. No detection curve should be drawn without actual labeled scores and a defined evaluation population. A passed internal test is evidence for its tested behavior only.

Next experiments

• Apply the protocol to one bounded integrity study and publish the complete case table.
• Add timing and resource measurements only with an explicit measurement environment and repeat count.

References

1. Vraelis evaluation guidance ↗
2. NIST adversarial machine learning taxonomy ↗
3. NIST AI Risk Management Framework ↗