Skip to content

Segment (Twilio Segment)

Segment — now Twilio Segment — is a customer data platform (CDP): a SaaS that ingests events from customer applications, normalises them to a common schema, and fans them out to hundreds of downstream destinations (warehouses, analytics tools, marketing tools, feature-flagging platforms). Engineering blog at segment.com/blog was active through the Twilio-acquisition transition; the AGENTS.md raw/_feeds.yaml notes "segment: now 404 after Twilio migration" — no live RSS feed, but individual posts remain reachable by URL.

Wiki anchor

The wiki's current anchor for Segment is the objects-pipeline changelog re-platforming story: a petabyte-scale DynamoDB table whose CDC changelog was migrated from Google Cloud BigTable to Amazon S3 for ~$0.6M/year of cost savings and infrastructure consolidation to a single cloud. The canonical framing that story crystallises on the wiki is changelog as secondary index — a CDC changelog exists to provide a query surface the base table's primary key does not, namely "items modified since timestamp T", and at petabyte scale the in-database answer (DynamoDB GSI) is priced out.

Key systems

  • systems/segment-objects-pipeline — the end-to-end service that ingests events, stores authoritative object state in a ~1 PB DynamoDB table (~958B items, growing), and maintains a CDC changelog feeding downstream warehouse integrations. V1 used BigTable as changelog store; V2 migrated the changelog to S3.

Key concepts and patterns

  • concepts/secondary-index — the CDC changelog's load-bearing job description, canonicalised from Segment's 2024-08-01 objects-pipeline post.
  • gsi-cost-anti-pattern-at-petabyte-scale — the cost-driven rejection of DynamoDB Global Secondary Index at petabyte base-table size.
  • cross-cloud-cost-consolidation — infrastructure consolidation to a single cloud as co-equal driver alongside storage-unit-cost delta.
  • object-store-as-cdc-log-store — S3 as the durable substrate for a CDC changelog feeding downstream batch consumers.

Recent articles

Last updated · 766 distilled / 2,225 read