Data integrity checks
A field guide for “Is this file intact and authentic?” when distributing or transferring data.
What to use when
Distinguish accidental corruption from checking whether a file came from the publisher you trust. Verification detects problems; it does not physically prevent changes.
Comparison
| Method | Purpose and conditions |
|---|---|
| Hash such as SHA-256 | Compare content with an expected digest. It does not establish origin if an attacker can replace that reference too. |
| Checksum such as CRC32 | Detect accidental transmission errors. Do not use it to defend against deliberate changes. |
| Digital signature | Verify signed content and its signer using a trusted key and signature scheme. Establish trust in the key or signer identity. |
Basic workflow for downloads/transfers
Reduce corruption and tampering risk with these steps.
Verify the downloaded file
- Obtain the expected digest for the exact version, platform and filename from the trusted publisher. HTTPS authenticates the connection to a domain, not the trustworthiness of the publisher itself.
- Hash the entire downloaded file with the same algorithm and compare it with the expected value. Do not run a mismatching file; check the artifact and download source.
- Follow the publisher's signature-verification procedure and check the expected key fingerprint or signer identity, as appropriate for that system. A valid signature alone does not identify your intended publisher.
- First compare the downloaded archive with the expected digest for that archive. To check extracted files, compare them with a separately trusted per-file manifest. Calculating a digest without a reference does not detect corruption.
Provide reference values and verification instructions
- Publish at least SHA-256; avoid MD5/SHA-1 alone.
- Provide signatures, verification instructions and a trusted way to establish the expected key fingerprint or signer identity. Finding a key on a key server or several matching websites does not by itself establish ownership.
- For file or chunk manifests, specify names, order, sizes and algorithms, and protect the manifest itself, for example with a signature. Concatenating chunk digests is not the same as hashing the complete file.
Choosing a hash
Balance security and speed.
Guidelines
- SHA-256 is widely used for releases. Match the publisher's algorithm and the exact bytes being checked.
- Follow the signature scheme, library and applicable policy. Digest length alone does not establish key trust or the security of the complete signature scheme.
- MD5/SHA-1 lack collision resistance—only keep them for legacy alongside a stronger hash.
Common pitfalls
Integrity requires trustworthy channels, not just hashes.
Checklist
- If an attacker can replace both the file and its reference digest, matching hashes are insufficient. Use a trusted reference channel or a signature verified with a trusted key.
- A normal file-content hash does not include its filename, modification time or permission bits. Changing line endings changes content bytes. Archives such as TAR can embed metadata, so metadata changes may change the archive's digest.
- An S3 multipart ETag is not the MD5 of the complete file. Encryption mode can also affect ETag semantics. For dedicated checksums, distinguish a full-object checksum from a composite checksum before comparing.
- Signature verification depends on trusting the public key (Web of Trust, pinned keys). A swapped key nullifies the check.
Takeaway
Match the expected value, input bytes and algorithm. Combine signature verification with trust in the key or signer identity. HMAC also authenticates using a shared secret, but all key holders can generate tags, unlike public signature verification. A valid signature does not prove software is harmless.
Sources and related guides
Python hashlib / Amazon S3 checksums / RFC 2104 / Hash fundamentals