A failure-injection matrix
Define faults, expected observations and untested combinations before running a campaign.
Research note · November 14, 2026Research question
How can a test suite cover meaningful boundaries without confusing case count with coverage?
Method
- List independent boundaries such as file read, parser, authorization, transport and runtime load. For each, define a bounded fault and the expected observable outcome.
- Run faults individually before combining them. Record which combinations are excluded because they obscure diagnosis or exceed the stated test boundary.
- Keep a case table with passed, failed, blocked and inconclusive outcomes. A blocked case should not disappear from the denominator or be counted as a successful rejection.
Records to retain
- Fault definitions and injection points
- Expected outcome per case
- Actual outcome and evidence
- Excluded combinations and rationale
Source basis
NIST’s AI RMF organizes evaluation around context and risk rather than a universal score. A fault matrix makes the relationship between a claim and its tested boundary inspectable.
Interpretation & limits
A large case count does not establish attack coverage or field reliability. Synthetic faults can miss interactions that appear in actual systems. No test campaign is claimed complete here.
Next experiments
- Choose one boundary and publish its complete case table.
- Review exclusions before expanding the number of cases.
References
Technical reference
Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.