Skip to content

SYSTEM Cited by 1 source

Amazon CloudWatch Metric Streams

Amazon CloudWatch Metric Streams is the push-based export primitive on top of Amazon CloudWatch: it streams CloudWatch metrics in near real-time to an Amazon Data Firehose delivery stream, with no customer code on the producer side. Metric Streams exists specifically to remove the pull-based failure modes of the Prometheus + CloudWatch-exporter shape — namely per-metric API throttling and the polling-cost + polling-latency tax — by inverting the data flow: CloudWatch pushes metric records into a delivery stream as they're recorded, and the delivery stream forwards them on (typically into S3, into another streaming sink, or into an HTTP endpoint).

Output formats

Metric Streams can emit metrics in three serialisation formats (per the 2026-05-13 AWS Architecture post):

  • OpenTelemetry 0.7 — legacy OTel-format, maintained for backwards compatibility.
  • OpenTelemetry 1.0 — current OTel-format; lets a Metric Streams → OTel-collector pipeline ingest natively without any format-translation step.
  • JSON — a self-describing CloudWatch-native envelope.

The architecture in the 2026-05-13 source explicitly chooses JSON, presumably because a Lambda transform sits between Firehose and the destination OTel collector and the function's logic is simpler over JSON than over the OTLP wire format.

Architectural role

Metric Streams sits at the production end of the observability fan-out: CloudWatch ingests metrics from every AWS service (default per-service metrics + Embedded Metric Format application metrics + custom PutMetricData), and Metric Streams is the export hatch into the broader streaming/observability ecosystem. Common shapes:

  • Metric Streams → Firehose → S3 for cheap long-term metric archival (the default Firehose-mandated destination).
  • Metric Streams → Firehose → HTTP endpoint for direct ingestion into a SaaS observability backend that supports the CloudWatch Streams partner integration (Datadog, New Relic, Splunk Observability Cloud, Honeycomb, etc.).
  • Metric Streams → Firehose → Lambda transform → private NLB → OTel collector in VPC — the VPC-private collector pattern from the 2026-05-13 source. The Lambda transform is needed because Firehose's HTTP endpoint destination is public-only, and the customer required the collector + metric data to remain VPC-internal.

Why streams are the answer to pull-based throttling

The 2026-05-13 source is explicit about the failure mode the push-based primitive was designed to fix. With pull-based monitoring (Prometheus + CloudWatch exporter), each metric the exporter scrapes generates a CloudWatch API call: at fleet scale the call volume saturates the API quotas, hits throttling, and drops metrics. "Our customer's current monitoring solution with Prometheus and Amazon CloudWatch exporter using a pull-based approach resulted in higher API throttling. This caused metric loss and created gaps in observability data for business-critical systems." Metric Streams replaces that polling traffic with a single producer-side push pipe — no per-metric API call — and delivers "sub-minute latency for real-time alerting" as a side benefit. See concepts/push-vs-pull-monitoring.

Seen in

  • sources/2026-05-13-aws-streaming-cloudwatch-metrics-to-vpc-based-opentelemetry-collectors-using-lambda — first canonical wiki naming. Metric Streams is the source-side primitive in the customer architecture: it streams CloudWatch metrics into a Firehose delivery stream in JSON format, which feeds a Lambda transform that pushes onto an internal NLB → OTel collector on EC2 inside the customer's VPC. The post canonicalises Metric Streams as the push-based replacement for pull-based Prometheus + CloudWatch-exporter, with the failure-mode rationale (API throttling under pull) and the architectural property (sub-minute, no per-metric API calls) explicit. Format support enumerated verbatim — OTel 0.7, OTel 1.0, JSON — with the customer-architecture choice (JSON) documented.
Last updated · 542 distilled / 1,571 read