Text encoding basics
When text garbles, it’s usually an encoding mismatch. This page outlines UTF-8 vs. UTF-16, BOM, and common pitfalls (like CSV encodings).
Key concepts
A character set maps characters to code points; an encoding turns code points into bytes.
UTF-8 vs. UTF-16
- Both encode Unicode; UTF-8 is variable-length, ASCII-compatible, and the Web default.
- UTF-16 uses 2 or 4 bytes per Unicode scalar value. A uses 1 byte in UTF-8 and 2 in UTF-16; U+3042 uses 3 and 2 respectively. Relative size depends on the text, excluding any BOM.
BOM (Byte Order Mark)
- UTF-8 BOM is 0xEF 0xBB 0xBF; optional but some tools expect it.
- A UTF-16 BOM signals byte order. Explicit UTF-16LE or UTF-16BE already specifies that order and does not require a BOM. Follow the format and receiver requirements.
- Whether a BOM is allowed depends on the format. RFC 8259 prohibits senders from adding a BOM to JSON transmitted over a network.
Common mojibake patterns
Garbled text appears when the reader assumes the wrong encoding.
Typical causes
- File saved as UTF-8 but opened as Shift_JIS/CP932 (common with some Excel defaults).
- Tools that expect BOM mis-handle BOM-less UTF-8, or treat BOM bytes as data.
- HTTP responses declaring the wrong charset (e.g., says Shift_JIS but sends UTF-8).
Fixes
- Preserve the original file and reopen it with the correct encoding before saving. Saving garbled text after replacement characters have been inserted can irreversibly lose information.
- Match the writer and reader encodings. Check conversion errors rather than silently replacing characters that the target encoding cannot represent.
- For HTML, match the HTTP encoding declaration to the actual bytes. The application/json media type defines no charset parameter, so do not assume every format uses the same declaration.
CSV pitfalls
CSV defaults vary by tool, so encodings often mismatch.
Encoding gotchas
- Some localized Excel versions default to CP932/Shift_JIS; UTF-8 CSV may garble.
- Microsoft documents normal opening for UTF-8 CSV with a BOM. For BOM-less files, import the data and select UTF-8. Menu names vary by version.
- Unix tools and data platforms expect UTF-8 (often BOM-less).
Recommended handling
- Use UTF-8 (no BOM) internally.
- For Excel users, provide BOM-ed UTF-8 or instructions to import with UTF-8.
- CSV has no general-purpose encoding declaration field; document the encoding separately. A BOM does not describe delimiters or column types.
Character counts and references
Characters, code points and bytes are different counts. A displayed character may contain multiple code points, for example combining marks or emoji sequences.