Tensor structure and byte integrity
Keep format validation separate from equality with a baseline.
Research note · November 14, 2026Research question
How can a file match a recorded digest and still be unusable as a tensor file?
Method
- Record a deliberately malformed fixture as its own baseline. A later byte comparison should match that baseline, even though an independent format parser rejects the file.
- Test header lengths, tensor offsets, declared shapes and truncated data regions independently. Preserve the exact parser error rather than reducing every failure to a generic changed-file status.
- Run the same fixture through the byte checker and the format validator. Present the two outcomes separately, including cases where validation cannot finish because of a resource limit.
Records to retain
- Fixture digest and length
- Format parser version
- Structural error category
- Independent integrity and validation outcomes
Source basis
Safetensors describes a bounded header and tensor offsets into a data buffer, with constraints on the buffer layout. These structural rules are independent of whether the file equals a previously recorded byte sequence.
Interpretation & limits
A structurally valid tensor file is not evidence of safe or accurate model behavior. Parser coverage and resource limits must be stated. No new validation benchmark is reported here.
Next experiments
- Publish minimal malformed fixtures without executable payloads.
- Check that user-facing results keep matching bytes distinct from valid structure.
References
Technical reference
Implementation details supporting this note, drawn directly from the maintained documentation. These are technical references, not additional papers or new experimental results. Expand a topic to inspect its method, fields and limits.