Skip to research
Vraelis Research
All researchResearch methods / Reference note and proposed protocol

A failure-injection matrix

Define faults, expected observations and untested combinations before running a campaign.

Research note · November 14, 2026

Research question

How can a test suite cover meaningful boundaries without confusing case count with coverage?

Method

  1. List independent boundaries such as file read, parser, authorization, transport and runtime load. For each, define a bounded fault and the expected observable outcome.
  2. Run faults individually before combining them. Record which combinations are excluded because they obscure diagnosis or exceed the stated test boundary.
  3. Keep a case table with passed, failed, blocked and inconclusive outcomes. A blocked case should not disappear from the denominator or be counted as a successful rejection.

Records to retain

  • Fault definitions and injection points
  • Expected outcome per case
  • Actual outcome and evidence
  • Excluded combinations and rationale

Source basis

NIST’s AI RMF organizes evaluation around context and risk rather than a universal score. A fault matrix makes the relationship between a claim and its tested boundary inspectable.

Interpretation & limits

A large case count does not establish attack coverage or field reliability. Synthetic faults can miss interactions that appear in actual systems. No test campaign is claimed complete here.

Next experiments

  • Choose one boundary and publish its complete case table.
  • Review exclusions before expanding the number of cases.

References

  1. NIST AI Risk Management Framework
  2. OpenTelemetry logs data model

Technical reference

Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.

Related notes