Skip to content

FLYIO 2024-02-15

Read original ↗

Fly.io — Globally Distributed Object Storage with Tigris

Summary

Fly.io's 2024-02-15 public-beta announcement for Tigris, a third-party globally distributed, S3-compatible object store integrated into Fly.io as the fly storage create primitive. Fly.io frames Tigris as the answer to "we didn't want to build S3-but-better" — a partner-built platform whose architecture is explicitly regional-first, not the usual "single-region bucket + CDN" shape that every framework lands you in. Architecturally: FoundationDB clusters in each Fly.io region track object metadata; Fly.io NVMe volumes serve as a first-level cached raw byte store; a queuing system modelled on Apple's QuiCK paper propagates object data to replicas, to regions where data is in demand, and out to third-party object stores including S3. Objects under ~128 KB are "instantly global" by default. The S3-compatible API means the AWS SDK and every framework S3 integration work unchanged. Commercially, Tigris usage rolls into the Fly.io bill (one invoice across compute, block storage, databases, networking, object storage).

Key takeaways

  1. "Single-write-region + CDN" is the default, but it's a bad fit for geographically distributed users. S3 was designed in the era when apps ran from one city — the bytes live in one specific data center, and latency grows with distance + with request frequency. Modern apps and their users live everywhere. "Routing those uploads and downloads through one building in Ashburn is no way to build a sandwich reviewing empire." The industry workaround — multiple regional buckets + CDN caching — complicates the application and puts barriers between it and its data. (Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

  2. CloudFront-style caching CDNs optimise the read side of a single-write-region bucket. Tigris is not that shape: it is "a toolset that you can use to build arbitrary CDNs, with consistency guarantees, instant purge and relay regions". "Globally distributed from the jump, with inter-region routing baked into its distribution layer." (Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

  3. Three-tier architecture: FoundationDB metadata + NVMe cache + queued distribution to replicas and third-party backends. "Tigris runs redundant FoundationDB clusters in our regions to track objects. They use Fly.io's NVMe volumes as a first level of cached raw byte store, and a queuing system modelled on Apple's QuiCK paper to distribute object data to multiple replicas, to regions where the data is in demand, and to 3rd party object stores… like S3." This is a concrete metadata-DB + object-cache-tier pattern: ACID-transactional metadata over eventually-replicated cached bytes, with a pluggable archival/origin tier (S3) behind. (Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

  4. Small objects (<128 KB) are instantly global by default. A size-threshold policy dictates when objects get proactively pushed across regions at write time vs. pulled on demand. Below the threshold, latency-sensitive reads anywhere hit local replicas. (Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

  5. Demand-driven replication to regions. Larger objects propagate to a region when they are accessed there, via the QuiCK-style queue. This is the demand-driven replication shape: the system learns where each object is hot and materialises local copies in response. (Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

  6. S3-compatible API as the drop-in interface. "If your framework can talk to S3, it can use Tigris." Fly.io's Tigris examples use the AWS libraries unchanged; fly storage create injects AWS_REGION / BUCKET_NAME / AWS_ENDPOINT_URL_S3 / AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY as app secrets. This is an instance of partner- managed service as native binding: a first-party CLI/config experience for a third-party platform. (Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

  7. Fly.io deliberately didn't build this themselves. "Compute and networking: those are things we love and understand. Object storage? We did not want to half-ass it. So we partnered with Tigris, so that they can put their full resources into making object storage as magical as Fly.io is." The article also frames billing unification as a first-class partnership feature: one bill for compute, block storage, databases (Supabase / PlanetScale), networking, and object storage (Tigris). See unified billing across providers. (Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

  8. S3 remains the archival-tier option behind Tigris. The distribution queue can propagate to "3rd party object stores… like S3" — i.e. Tigris can be configured to use S3 as an origin/archival destination rather than replacing it. The S3-compatible API on the front means existing S3 code keeps working; S3-as-backend on the back means existing S3 data/tooling integration keeps working too. (Source: sources/2024-02-15-flyio-globally-distributed-object-storage-with-tigris)

Systems extracted

  • systems/tigris — the subject; S3-compatible globally distributed object store built on FoundationDB + Fly.io NVMe + QuiCK-style queuing.
  • systems/foundationdb — "redundant FoundationDB clusters in our regions to track objects" — Tigris's metadata layer, regionally replicated.
  • systems/quick-queue — the Apple QuiCK paper's queuing model, used here to distribute object bytes across replicas, demand regions, and third-party backends.
  • systems/aws-s3 — named as both the incumbent Tigris improves on AND as a pluggable third-party backend Tigris can archive to.
  • systems/amazon-cloudfront — named as the "Cloudfront caching CDN" shape Tigris is explicitly not — a single-write- region + read-cache architecture rather than true global distribution.
  • systems/firecracker — Fly.io's underlying isolation layer ("transmute containers into VMs, running them on our hardware around the world with the power of Firecracker alchemy"). Context only; not central to the Tigris architecture.

Concepts extracted

  • concepts/demand-driven-replication — Tigris propagates object bytes to regions where they are in demand, rather than mirroring every object to every region on write. New concept page — this is the first wiki instance.
  • concepts/immutable-object-storage — Tigris presents the S3 immutable-objects contract (PUT/GET/DELETE/LIST, whole-object replacement).
  • concepts/geographic-sharding — metadata + bytes are partitioned by region; reads are served from the nearest region holding a copy.
  • concepts/edge-to-origin-database-latency — the "Ashburn round-trip" problem for apps whose users aren't in Virginia but whose S3 bucket is.
  • concepts/egress-cost — CDN / cross-region replication decisions are shaped by the egress-per-byte cost line, even when not explicitly quoted in this piece.
  • concepts/file-vs-object-semantics — Tigris keeps the object contract (not a filesystem), so existing S3 code and notification pipelines port cleanly.

Patterns extracted

  • patterns/metadata-db-plus-object-cache-tier — Tigris's three-tier shape: strongly-consistent metadata in FoundationDB, eventually-consistent cached bytes in NVMe, pluggable origin tier behind (S3 etc.). New pattern — first wiki instance.
  • patterns/caching-proxy-tier — NVMe-on-Fly.io acts as a regional cache tier in front of the distributed byte store; shape-adjacent to the cache-proxy pattern wiki already has (FigCache / Redis).
  • patterns/partner-managed-service-as-native-binding — fly storage create makes a third-party service look like a first-party Fly.io primitive, with secrets injected into the user's app automatically.
  • patterns/unified-billing-across-providers — Fly.io rolls Tigris (and Supabase / PlanetScale / Upstash) usage into the Fly.io bill so the customer has one monthly invoice.

Operational numbers

  • Object-size threshold for instant global replication: ~128 KB. Below this, objects are "instantly global… by default"; above, demand-driven.
  • Setup time claim: single-digit minutes. fly storage create → bucket provisioned + five S3-compatible env vars injected as app secrets. Fly.io frames this against its two-hour "full-stack developer viability" threshold.
  • Comparable (CloudFront): "probably within 2 hours" to set up an S3 + CloudFront caching CDN — Fly.io's stated baseline for the competing shape.

No latency numbers, throughput, durability class, replica count, region count, or consistency guarantees are quoted — this is a partnership announcement, not an architecture deep-dive.

Caveats

  • This is a partnership-announcement post, not a Tigris architecture paper. The three-tier architecture statement (FDB metadata + NVMe cache + QuiCK queuing + S3 backend) is the only technical disclosure, and it is one sentence.
  • No consistency-guarantees disclosure. Tigris is called "a toolset that you can use to build arbitrary CDNs, with consistency guarantees" — but what those guarantees are (read- after-write? regional strong consistency with cross-region eventual? per-object quorum?) is not stated.
  • No replication-policy disclosure. "Redundant FoundationDB clusters" — how many, across how many regions, replication strategy (Raft/Paxos layer? sync vs async cross-region?) is not discussed.
  • No durability class disclosure. Whether Tigris targets the S3 "eleven nines" class, or a lower tier, is not stated.
  • No NVMe-cache eviction/retention policy disclosure. Size threshold (~128 KB) for instant-global is the only policy hint; what happens to colder objects on NVMe-exhausted regions is undisclosed.
  • Fly.io tier-3 source. Treated per AGENTS.md Tier-3 selectivity: accepted on scope because it covers distributed- systems internals (multi-region object-store architecture), even though the piece is framed as a product-launch announcement.
  • Tigris is a third-party company (Tigris Data, Inc.) — not a Fly.io-built system. The architectural statements come from Fly.io's description of Tigris, filtered through a partnership-announcement narrative.

Contradictions / open questions

  • None directly contradicting existing wiki content. The piece extends existing systems/foundationdb framing (new production use case: metadata-for-global-object-store; prior wiki instance was Datadog Husky's fragment metadata) and existing systems/aws-s3 / concepts/immutable-object-storage context (Tigris as an S3-compatible-API front over a different-shaped backend).
  • QuiCK paper citation is worth a dedicated stub even though details aren't disclosed — the queuing-model lineage (Apple / FoundationDB team) is itself useful wiki content.

Source

Last updated · 178 distilled / 1,178 read