SYSTEM Cited by 1 source

Zstandard (zstd)

Zstandard (zstd) is Meta's open-source general-purpose lossless compressor, announced 2016. Pitch at the time (retold in the 2025 OpenZL post): "the same or better compression ratio of prior default but at the much increased speed required by datacenter workloads. By pairing strong entropy coding with a design that fully utilized modern CPU capabilities, Zstandard offered a substantial improvement that justified its presence in datacenters." (Source: sources/2025-10-06-meta-openzl-an-open-source-format-aware-compression-framework.)

Project

• GitHub: facebook/zstd
• Original 2016 announcement: engineering.fb.com — Smaller and faster data compression with Zstandard — not yet ingested on the wiki (referenced in the OpenZL post).
• Role on this wiki: stub. Entry exists to anchor the OpenZL post's references + the Managed Compression lineage.

Relationship to OpenZL

OpenZL is positioned by Meta as Zstandard's architectural successor for structured data — not a replacement for Zstandard as a general-purpose compressor. Specifically:

1. zstd is OpenZL's speed + ratio baseline. The Silesia sao headline numbers are zstd -3 vs xz -9 vs OpenZL (Source: sources/2025-10-06-meta-openzl-an-open-source-format-aware-compression-framework).
2. zstd is OpenZL's fallback. When OpenZL doesn't understand the input (pure text, unknown format), it "falls back to zstd, offering essentially the same level of performance." This is the fallback-to-zstd pattern — OpenZL's worst case is zstd-equivalent, not worse.
3. zstd dictionaries are the original Managed Compression workload. Meta originally built Managed Compression "to automate dictionary compression with Zstandard"; OpenZL Plans extend the same lifecycle to format-aware compression.

Architecture (sketch)

OpenZL's post names zstd's architecture only at the level of "strong entropy coding" + "design that fully utilized modern CPU capabilities." Deeper zstd architecture (LZ77 matching, FSE / Huffman entropy coding, dictionary support, compression levels, streaming API) is not described in the source currently ingested on the wiki; deepening this page is expected when the 2016 Zstandard announcement or any later zstd-internals post is ingested.

Where zstd sits on the wiki

• Data-center general-purpose compression → zstd is the default.
• Per-collection storage compression → zstd is a WiredTiger blockCompressor option alongside snappy + zlib (Source: MongoDB Cost of Not Knowing Part 3).
• Format-aware compression → delegated to OpenZL, with zstd as fallback.

Seen in

• sources/2025-10-06-meta-openzl-an-open-source-format-aware-compression-framework — Zstandard is the speed + ratio baseline, the fallback engine, and the lineage Meta invokes for motivating OpenZL.

Related

• systems/openzl — OpenZL is zstd's architectural successor for structured data; falls back to zstd when no structure is available.
• systems/managed-compression-meta — zstd dictionary training was the original Managed Compression use case.
• concepts/format-aware-compression — the category OpenZL defines against generic compressors like zstd.
• patterns/fallback-to-general-purpose-compressor — zstd as fallback inside OpenZL.
• companies/meta