Choosing data formats

“Just use CSV or JSON” is common, but each format has sweet spots. Here’s a quick guide for exchange, storage, and analytics.

When you need tabular data

For rows and columns, pick text or columnar binary based on how you’ll process it.

Human-readable text. CSV uses commas; TSV uses tabs and is simpler when values contain commas.
RFC 4180 CSV quotes fields containing commas, newlines or double quotes; an embedded double quote is doubled. Agree on headers, encoding and line endings because implementations vary. For TSV, agree on handling embedded tabs and newlines.
Great for ad hoc exchange and spreadsheets; you still need to share the schema separately.

Parquet is a column-oriented storage format with types, compression and encodings. It suits analytics that read selected columns; performance depends on data, settings and workload.
Arrow primarily specifies columnar in-memory representation and data interchange. It also defines IPC stream and file formats. Arrow and Parquet serve different roles.
Check the types, versions and compression supported by the consuming tools. Long-term storage also requires schema management and software that can continue to read the data.

For nested objects and arrays, JSON variants are a good fit.

Key/value with nesting; common for APIs and configs.
JSONL uses UTF-8 and one JSON value per line, including arrays or null, not only objects. Blank lines are invalid. Read multiple lines as separate values rather than as a single JSON document.
JSON has strings, numbers, booleans, null, arrays and objects, but no date-specific type. Agree on required fields, date representations and numeric precision; validate with a schema where useful.

Balance readability, size, speed, and tool support.

Human-friendly editing: TSV/CSV (mind escaping).
APIs, configs, logs: JSON/JSONL (manage schema separately).
Analytical storage: Parquet. Column data interchange during processing: Arrow. Benchmark actual data with the intended tools before choosing.

< Data size units Text encoding basics >