Skip to content

SYSTEM Cited by 2 sources

Zstandard (zstd)

Zstandard (zstd) is Meta's open-source general-purpose lossless compressor, announced 2016. Pitch at the time (retold in the 2025 OpenZL post): "the same or better compression ratio of prior default but at the much increased speed required by datacenter workloads. By pairing strong entropy coding with a design that fully utilized modern CPU capabilities, Zstandard offered a substantial improvement that justified its presence in datacenters." (Source: sources/2025-10-06-meta-openzl-an-open-source-format-aware-compression-framework.)

Project

Relationship to OpenZL

OpenZL is positioned by Meta as Zstandard's architectural successor for structured data — not a replacement for Zstandard as a general-purpose compressor. Specifically:

  1. zstd is OpenZL's speed + ratio baseline. The Silesia sao headline numbers are zstd -3 vs xz -9 vs OpenZL (Source: sources/2025-10-06-meta-openzl-an-open-source-format-aware-compression-framework).
  2. zstd is OpenZL's fallback. When OpenZL doesn't understand the input (pure text, unknown format), it "falls back to zstd, offering essentially the same level of performance." This is the fallback-to-zstd pattern — OpenZL's worst case is zstd-equivalent, not worse.
  3. zstd dictionaries are the original Managed Compression workload. Meta originally built Managed Compression "to automate dictionary compression with Zstandard"; OpenZL Plans extend the same lifecycle to format-aware compression.

Architecture (sketch)

OpenZL's post names zstd's architecture only at the level of "strong entropy coding" + "design that fully utilized modern CPU capabilities." Deeper zstd architecture (LZ77 matching, FSE / Huffman entropy coding, dictionary support, compression levels, streaming API) is not described in the source currently ingested on the wiki; deepening this page is expected when the 2016 Zstandard announcement or any later zstd-internals post is ingested.

Where zstd sits on the wiki

  • Data-center general-purpose compression → zstd is the default.
  • Per-collection storage compression → zstd is a WiredTiger blockCompressor option alongside snappy + zlib (Source: MongoDB Cost of Not Knowing Part 3).
  • Format-aware compression → delegated to OpenZL, with zstd as fallback.
  • CDN cache-at-rest compression → zstd is the codec behind Cloudflare's Cache Transcoding prototype (see below).

As Cloudflare's Cache Transcoding codec (2026-09-01)

Cloudflare's Cache Transcoding prototype uses zstd to compress eligible uncompressed cache text at rest inside its Pingora proxy, keeping the compressed form across Tiered Cache and decoding only on the client-facing hop. Concrete zstd datapoints from that post:

  • Level 3 chosen as the speed/ratio default so cache fills don't become a CPU bottleneck (level is a tunable parameter).
  • ~2.834× compression ratio on their (deliberately compressible) test corpus.
  • Encode 4.31 ns/byte (~232 MB/s); decode 1.56 ns/byte (~641 MB/s) — the encode/decode asymmetry that underwrites the encode-once, decode-many economics.
  • Cloudflare's own comparison: zstd compressed 42% faster than Brotli at near-equal size, and produced files 11.3% smaller than gzip at comparable speed — the basis for choosing zstd for a latency-sensitive, large-traffic-fraction feature.

(Source: sources/2026-09-01-cloudflare-how-we-could-save-petabytes-of-cache-storage-with-zstandard)

Seen in

Last updated · 766 distilled / 2,225 read