Segment (Twilio Segment)¶
Segment — now Twilio Segment — is a customer data platform (CDP): a SaaS that ingests events from customer applications, normalises them to a common schema, and fans them out to hundreds of downstream destinations (warehouses, analytics tools, marketing tools, feature-flagging platforms). Engineering blog at segment.com/blog was active through the Twilio-acquisition transition; the AGENTS.md raw/_feeds.yaml notes "segment: now 404 after Twilio migration" — no live RSS feed, but individual posts remain reachable by URL.
Wiki anchor¶
The wiki's current anchor for Segment is the objects-pipeline changelog re-platforming story: a petabyte-scale DynamoDB table whose CDC changelog was migrated from Google Cloud BigTable to Amazon S3 for ~$0.6M/year of cost savings and infrastructure consolidation to a single cloud. The canonical framing that story crystallises on the wiki is changelog as secondary index — a CDC changelog exists to provide a query surface the base table's primary key does not, namely "items modified since timestamp T", and at petabyte scale the in-database answer (DynamoDB GSI) is priced out.
Key systems¶
- systems/segment-objects-pipeline — the end-to-end service that ingests events, stores authoritative object state in a ~1 PB DynamoDB table (~958B items, growing), and maintains a CDC changelog feeding downstream warehouse integrations. V1 used BigTable as changelog store; V2 migrated the changelog to S3.
Key concepts and patterns¶
- concepts/changelog-as-secondary-index — the CDC changelog's load-bearing job description, canonicalised from Segment's 2024-08-01 objects-pipeline post.
- concepts/gsi-cost-anti-pattern-at-petabyte-scale — the cost-driven rejection of DynamoDB Global Secondary Index at petabyte base-table size.
- concepts/cross-cloud-cost-consolidation — infrastructure consolidation to a single cloud as co-equal driver alongside storage-unit-cost delta.
- patterns/object-store-as-cdc-log-store — S3 as the durable substrate for a CDC changelog feeding downstream batch consumers.