Skip to content

SYSTEM Cited by 1 source

Town Lake (Cloudflare data platform)

Town Lake — named after its namesake reservoir in Austin, Texas — is Cloudflare's unified data analytics platform: a data lakehouse with a single SQL interface to everything Cloudflare knows. Introduced publicly in the 2026-05-28 launch post alongside its AI agent Skipper.

What it replaces

Pre-Town Lake, Cloudflare had the canonical hyper-growth data-sprawl shape:

  • Postgres for account metadata.
  • ClickHouse for analytics events.
  • BigQuery for usage rollups.
  • R2 for raw logs.
  • Kafka topics for real-time signals.
  • A long tail of pipelines stitching them.

Each tier had its own credentials, query language, and retention policy. "To answer a simple question like 'How many domains that signed up today are in the Top 100 by traffic?', an analyst at Cloudflare had to know which system to ask, what credentials to use, what query language to write, and whether the data they were looking at was sampled, fresh, or seven-days stale." Parts of the analytics-pipeline downsample to handle 700M+ events/second — "the right behavior when you want an analytics dashboard to load, but it's exactly the wrong behavior when you are trying to compute someone's usage required to issue an invoice." Plus an external dependency on another cloud for some critical data.

Architecture

Town Lake's architecture is the standard lakehouse decomposition with five named in-house subsystems and one open-source query engine:

Component Role
Apache Trino Query engine — federated SQL across Postgres, ClickHouse, R2/Iceberg, BigQuery
R2 Data Catalog Managed Iceberg service holding cold/warm data on R2
DataHub Metadata catalog — every table, column, owner, lineage edge, glossary term
Lifeguard Access control — D1 + internal IAM → JSON policy served to Trino over HTTP
Skimmer Continuous PII detection — fast classifier + agentic second pass
Transformer ELT engine — YAML-frontmatter SQL DAG on Workflows with DO state
Ingestion Long-lived k8s orchestrator + short-lived worker jobs (Postgres / ClickHouse → Parquet → Iceberg on R2)

The single-engine property is the architectural promise: "a single SQL query can join a Postgres table, a ClickHouse table, and an Iceberg table on R2 without a need to materialize the intermediate results into a different system. A query that asks 'what are the top 100 paying customers by Workers requests this week' compiles into a plan that pushes filters into ClickHouse, joins against an account dimension in Postgres, and ranks against billing rollups in R2, all in one go." Canonicalised at patterns/single-sql-interface-over-heterogeneous-sources.

Built on Cloudflare's customer Developer Platform

The post's deliberate framing: "if we were going to make a major investment in our data infrastructure, it was going to be built on the same products we sell to customers."

  • R2 — storage tier (Iceberg tables + Parquet files).
  • Workers — compute primitive for Skipper and the embed iframe surface.
  • Cloudflare Access — authentication; gates iframe contents on shared dashboards.
  • Workflows — orchestration substrate for the Transformer ELT engine.
  • Durable Objects — per-DAG state machine for Transformer.
  • D1 — relational metadata for Lifeguard rules + Transformer run history.
  • KV — ephemeral caches.
  • Workers AI — Skimmer's PII classifier primitive.

This is the most comprehensive dogfood-as-reference-architecture post in Cloudflare's corpus to date — every Developer Platform primitive threading through one workload at billion-events/second scale.

Governance shape: default-closed

The structural inversion versus traditional analytics platforms:

  • Traditional: open-by-default, restrict-by-exception. Allow access to everything, audit and lock down sensitive tables when someone notices.
  • Town Lake: tables are inaccessible until reviewed. New databases connected to Trino are scanned by Skimmer, columns classified, registered in the central allowlist as pending. Until a reviewer approves the table (and the specific columns within it), users can't query it.

Two affordances make this operationally tractable:

  1. Skimmer's classifier is reasonably good — "catches obvious PII (emails, IPs, names, phone numbers) and the long tail of non-obvious sensitive data (API tokens that match certain prefixes, opaque IDs that can be traced back to users). Reviewers see what was detected and either approve, override, or deny. Most reviews take seconds."
  2. Self-serve workflow — query errors are "this table needs review, click here to request one" not "permission denied". Skipper "will even suggest the right RBAC group to request and link you straight to it."

Canonicalised at concepts/default-closed-table-allowlist + patterns/default-closed-allowlist-with-automated-pii-scan.

Storage tiering

Iceberg's recompaction pipeline turns recency into the storage-cost dial:

  • Per-minute usage — last week.
  • Hourly — last quarter.
  • Daily — beyond.

"The storage cost decreases as recency does, while the data stays queryable. Parquet files in R2 are much cheaper compared to keeping the same data in an OLAP database."

Operational scope

  • >1 billion events per second on Cloudflare's network (the motivating scale).
  • 53% of all Town Lake queries are billing-related; 91,760 queries from 324 distinct Cloudflare employees in a recent measurement period.
  • Legacy revenue-rollup SQL: 200–300 lines → now 5 lines.
  • Top-customer / top-domain conversational queries land in ~3 seconds through Skipper.

Future direction

The post names two paths:

  • Deeper integration with internal chat + ticketing systems so "ask the data" becomes "the natural first move for anyone debugging an incident, scoping a project, or sanity-checking a hypothesis."
  • Migrating Town Lake's workflow over to R2 SQL — Cloudflare's serverless distributed analytics query engine — "as its feature set expands, we plan to move many parts of Town Lake's workflow over to it." Trino remains the engine today; R2 SQL is the named successor.

Seen in

Last updated · 542 distilled / 1,571 read