Skip to content

SYSTEM Cited by 2 sources

Amazon S3 Files

S3 Files (launched 2026-04-07) is an S3 feature that makes any S3 bucket or prefix mountable as an NFS filesystem on EC2 instances, containers (ECS / EKS), and Lambda functions. The wire protocol is NFS v4.1+, full compliance (create / read / update / delete). It is backed by Amazon EFS under the covers and delivers ~1 ms latencies for active data. From the application's perspective, it's a mounted directory; from S3's perspective, the data is objects in a bucket. The key design commitment is that the file/object boundary is explicit and programmable, not hidden.

(Sources: sources/2026-04-07-allthingsdistributed-s3-files-and-the-changing-face-of-s3, sources/2026-04-07-aws-s3-files-mount-any-s3-bucket-as-a-nfs-file-system-on-ec2-ecs)

Deployment model: mount target as VPC network endpoint

A mount target is a VPC network endpoint that sits between compute (EC2, ECS, EKS, Lambda) and the S3 file system — the bridge through which NFS traffic reaches the file system. Console auto-creates mount targets; CLI requires two separate commands:

aws s3 create-file-system ...
aws s3 create-mount-target ...

Clients mount with two shell lines:

sudo mkdir /home/ec2-user/s3files
sudo mount -t s3files fs-<id>:/ /home/ec2-user/s3files

Files created via standard Unix tools (echo "Hello S3 Files" > s3files/hello.txt) appear in S3 shortly after (aws s3 ls s3://bucket/hello.txt) with identical content. GA in all commercial AWS Regions at launch (2026-04-07).

Why it exists

The friction shape is described in Warfield's UBC genomics origin story: analytical tools (GATK4, ML pretraining, pandas, build systems, log processors, design tools, agentic dev tools) expect a local Linux filesystem; data lives in S3; users write copy pipelines in the gap. Agents amplify this — a coding agent reasoning over a dataset must add "list S3 → copy local → operate" to its chain; filesystem access collapses that.

Design journey: why "EFS3" failed

The team started in 2024 with the obvious convergence goal — fuse EFS and S3 into one unified system. Six months of whiteboard sessions produced a "battle of unpalatable compromises": every path required giving up either file or object semantics on the merged system. The team framed the enumerated compromises as "the lowest common denominator, not the best of both worlds" — example workloads broke in surprising ways on both sides.

Post-Christmas-2024 pivot: stop trying to hide the boundary; make the boundary the feature. File access becomes a presentation layer over S3. See concepts/boundary-as-feature, patterns/presentation-layer-over-storage.

Architecture (stage and commit)

Borrowed from version control. Two synchronised namespaces under one logical mount:

  • EFS namespace (file view) — NFS semantics, directory ops, permissions on the mount; provides close-to-open consistency.
  • S3 namespace (object view) — full S3 atomic-PUT strong consistency, notifications, CRR/replication, IAM, existing object semantics untouched.
  • Sync layer — bidirectional; file-side changes accumulate, commit back to S3 as a single PUT per changed object roughly every 60 seconds; external S3 mutations show up in the filesystem view automatically.

See concepts/stage-and-commit.

Mount behaviour and lazy hydration

  • On first access to a directory, S3 Files imports metadata from S3 to populate a file-optimised namespace, as a background scan.
  • Files < 128 KB hydrate fully (metadata + data) on that import.
  • Larger files hydrate metadata only; data fetched on first read.
  • Effect: mount a bucket with millions of objects and start working immediately (no blocking prefetch). See concepts/lazy-hydration.

Conflict policy

  • Concurrent same-file edit from both sides → S3 wins as source of truth; the filesystem-side version moves to a lost+found directory; a CloudWatch metric fires.
  • Asymmetric by design: S3 object semantics are the preservation target; file-side loss is visible (metric + directory), not silent.

Eviction and cost

  • File data unused for 30 days is evicted from the filesystem view (not from S3).
  • Storage cost on the file side stays proportional to the active working set.

Read bypass (throughput escape hatch)

  • High-throughput sequential reads reroute off NFS to parallel direct-GETs against S3.
  • Reported: 3 GB/s per client (with room to improve), terabits per second across many clients.
  • The same throughput tier CRT-based S3 clients already get, without leaving the filesystem API. See systems/aws-crt.
  • Byte-range reads transfer only the requested bytes — minimises data movement and cost for random-access over large files.
  • Intelligent pre-fetching anticipates access needs; customer controls whether to load full file data or metadata only per file.

Performance-tier split (high-performance storage vs direct-S3)

S3 Files internally routes file access between two physical-storage tiers:

  • High-performance storage tier (EFS-backed) — files that benefit from low-latency access are stored and served here. ~1 ms latencies for active data. NFS close-to-open consistency supports concurrent access from multiple compute resources.
  • Direct-S3 serving — files needing large sequential reads are served directly from Amazon S3 to maximize throughput, bypassing the filesystem-storage tier. This is how S3 Files keeps the full S3 throughput envelope for bulk-read workloads without the customer leaving read(2).

Together with byte-range reads + intelligent pre-fetching, this is how S3 Files matches workload shape to physical-storage location automatically.

Four hard subproblems and how each was resolved

Each became an asymmetric, explicit decision rather than a convergent compromise. See patterns/explicit-boundary-translation for the generalised shape.

Consistency and atomicity

  • Filesystems assume atomic rename and atomic directory rename as primary change-visibility primitives.
  • S3 has no native rename; conditional PUT helps partially, nothing analogous for directory rename.
  • Resolution: don't unify. File-side mutations/renames happen freely; S3-side sees the resulting whole-object PUTs on commit; NFS close-to-open consistency for file readers, S3 atomic-PUT strong consistency for object readers.

Authorization

  • IAM: rich (prefix-scoped, network-conditioned, request-property- conditioned), expensive to evaluate.
  • Filesystem perms: inode-attached, directory-traversal-gated, handle-cached; cheap; complicated by hard links.
  • Resolution: permissions specified on the mount and enforced in the filesystem; IAM retained as backstop (perimeter disable at any time); mappings defined at the mount boundary; "the door open" for multiple mounts with different auth configs over the same data.

Namespace semantics — the "dreadful incongruity"

  • Filesystems have first-class / as path separator.
  • S3's / is a suggestion — LIST accepts any delimiter; customers build multi-dimensional naming w/ multiple embedded delimiters.
  • S3 allows an object key ending in / — looks like a directory but is an object ("filerectories," thankfully not kept).
  • Resolution: each side keeps its native naming. When an object or file can't be represented on the other side, emit an event so customers can monitor and act. "Downloading complexity onto the developer" framed as the right call because the alternative is silently breaking applications in their native domain.

Namespace performance asymmetry

  • File systems: directory-co-located metadata, traversal-heavy access patterns; fast distributed file namespaces co-locate per-dir metadata on one host to keep lookups fast.
  • Object systems: flat namespace optimised for highly-parallel point queries; "individual 'directories' have billions of objects in them and are being accessed by hundreds of thousands of clients in parallel."
  • Resolution: maintain both namespaces (EFS's + S3's), optimised for their native workloads, with the sync layer between them. Don't try to make one serve the other's access pattern.

Known limitations (launch-day)

  • Renames are expensive. S3 has no native rename → renaming a directory copies + deletes every object under that prefix. A mount warning fires when mount covers > 50M objects for this reason.
  • No explicit commit control yet. The 60-second commit window is fixed at launch; "works for most workloads but we know it won't be enough for everyone." Future work on the stage-and-commit programmable surface.
  • Some S3 keys cannot be represented as valid POSIX filenames — those won't appear in the filesystem view; an event fires. See concepts/file-vs-object-semantics for the full list of incongruities.
  • ~9 months of customer beta shaped the above edges; they're called out up-front rather than papered over.

The "multiphase, not concurrent" insight

The design commitment that made adoption possible:

"It turns out that very few applications use both file and object interfaces concurrently on the same data at the same instant. The far more common pattern is multiphase. A data processing pipeline uses filesystem tools in one stage to produce output that's consumed by object-based applications in the next."

So the goal isn't concurrent-coherent file+object semantics — it's same data in one place, with the right view for each access pattern, plus synchronisation. This is the architectural move that keeps the existing "enormous numbers of existing buckets serving applications that depend on S3's object semantics working exactly as documented" undisturbed.

Positioning vs Amazon FSx

AWS carves up the managed-filesystem space explicitly at S3 Files launch:

  • S3 Files: interactive + shared access to S3-resident data through a high-performance file interface. Target workloads — multi- compute-resource collaboration over S3 data, agentic AI using Python libraries and CLI tools, ML training pipelines mutating datasets. Shared access across clusters without duplication, sub-millisecond latency, automatic S3 sync.
  • Amazon FSx: better for on-prem NAS migration, HPC + GPU cluster storage (FSx for Lustre), or specific filesystem-family capabilities (FSx for NetApp ONTAP, OpenZFS, Windows File Server).

Principle: S3 Files makes S3 itself the data hub accessible from any AWS compute — it's not aimed at replacing on-prem NAS migration.

Pricing model (four line items)

  • Data stored in the S3 file system (high-performance tier).
  • Small-file read operations on the file system.
  • All write operations on the file system.
  • S3 requests incurred during sync between filesystem and bucket.

Large-file reads served direct from S3 fall under normal S3 GET pricing, consistent with the read-bypass design. Reference: S3 pricing page.

Place in S3's data-primitive lineage

S3 Files is the third first-class data primitive added to S3's platform surface in 18 months:

  1. Objects (2006, origin) — systems/aws-s3.
  2. Tables (re:Invent 2024) — systems/s3-tables, managed Iceberg.
  3. Vectors (re:Invent 2025) — systems/s3-vectors, elastic similarity-search indices.
  4. Files (2026-04-07) — this system.

"Different ways of working with data aren't a problem to be collapsed. They're a reality to be served." Each primitive targets the same S3 property floor (elasticity, durability, availability, performance, security) presented in the shape that fits the workload. See systems/aws-s3 for the full framing.

Seen in

Last updated · 200 distilled / 1,178 read