Skip to content

PATTERN Cited by 1 source

Firehose Lambda transform as VPC-private endpoint bridge

Amazon Data Firehose is the managed "push streaming records to a destination" primitive on AWS. It supports a generic HTTP endpoint destination — but with a critical constraint: the HTTP endpoint must be public. Firehose will not deliver into a private VPC endpoint. This is a structural architectural mismatch any time the receiving service is intentionally VPC-internal (data-privacy, regulatory, or network-topology reasons).

The Firehose Lambda transform as VPC-private endpoint bridge pattern resolves the mismatch using Firehose's existing data-transformation hook: the transform Lambda runs with VPC attachments, receives Firehose-buffered records synchronously, and pushes them onto the private VPC endpoint itself, taking Firehose's place at the HTTP-egress layer. Firehose still drives the pipe; Lambda is just the bridge that crosses the public/private boundary on Firehose's behalf.

The constraint that motivates the pattern

From the 2026-05-13 source:

"Amazon Data Firehose natively supports data delivery to HTTP endpoints, but these endpoints must be public — they cannot be private endpoints inside a VPC. To overcome this limitation, we use the Amazon Data Firehose transform configuration, that invokes a Lambda function synchronously, which then securely pushes the metrics through the NLB endpoint to the collector running within the VPC." (Source)

The customer's data-privacy requirement was that "the metric data and the OpenTelemetry collector to be within their VPC." The Firehose HTTP endpoint destination simply could not satisfy that constraint, so the architecture re-purposes Firehose's transformation feature — normally meant for record-level preprocessing — as the actual delivery mechanism.

Mechanism

┌────────────────────────┐        ┌──────────────────────────────┐
│ CloudWatch / Producer  │ push   │  Amazon Data Firehose         │
└─────────┬──────────────┘  ────▶ │  (delivery stream, buffered)  │
          ▼                        └──────────────┬───────────────┘
                                                  │ sync invoke
                                                  ▼
                                       ┌────────────────────────┐
                                       │ Lambda transform fn     │
                                       │ (VPC-attached)          │
                                       └─────────┬──────────────┘
                                                 │ HTTP push
                                                 ▼
                              ┌────────────────────────────────────┐
                              │ Internal NLB                        │
                              │ (private subnets, VPC-internal)     │
                              └────────────────┬───────────────────┘
                                               ▼
                              ┌────────────────────────────────────┐
                              │ Target service                      │
                              │ (e.g. OTel collector on EC2)        │
                              └────────────────────────────────────┘

The Lambda function:

  1. Is configured as Firehose's data-transformation function on the delivery stream.
  2. Has VPC attachments (subnets + security groups) so its network egress originates inside the customer's VPC.
  3. On synchronous invocation, receives a Firehose batch of buffered records.
  4. POSTs the records (translated to the target endpoint's wire format if needed) to the private NLB inside the VPC.
  5. Returns success / failure to Firehose; Firehose retries on failure per its at-least-once contract.

The Firehose stream still has S3 configured as the canonical fallback destination — but in steady state no records reach S3 because the Lambda transform delivers them all.

Why this works (and where it differs from async event-source mapping)

The pattern is non-obvious because Lambda + AWS streaming usually means async event-source mapping (Kinesis Data Streams → Lambda, SQS → Lambda, DynamoDB Streams → Lambda) — Lambda consumes the records, returns success/fail, and the upstream advances. Firehose data transformation is a different invocation contract: synchronous, per record batch, return value is the delivery payload. The architecture re-interprets that contract:

  • Normal use: transform mutates records, Firehose delivers the return value to its configured destination.
  • This pattern's use: Lambda performs the delivery itself to the VPC-internal endpoint, and returns a no-op / pass-through payload that Firehose's destination (S3 fallback or "no delivery") then sinks.

Trade-offs

Property Native Firehose HTTP endpoint Lambda transform bridge
Destination address Must be public DNS Any VPC-internal endpoint
Delivery latency Direct from Firehose Adds Lambda invocation + cold start
Failure modes Firehose retry, then S3 fallback Lambda retry + Firehose retry, then S3 fallback
Cost Firehose pricing only Firehose + Lambda invocations + Lambda duration
Operational surface Configure destination URL Author + maintain Lambda code
Security posture Requires public endpoint All metric data stays in VPC
At-least-once semantics Yes, Firehose retries Yes, but with two-tier retry (Lambda + Firehose)

The pattern's primary cost is synchronous Lambda invocation latency on the critical path. Cold starts can stretch the delivery deadline; if the Lambda is VPC-attached it pays the historical VPC-cold-start tax (now greatly mitigated by Lambda's post-2026 networking retrofit disclosed in the 2026-04-22 Invisible Engineering post). The pattern's primary benefit is all metric data stays inside the customer's VPC, which for some compliance regimes is the critical property.

Generalisations

The pattern generalises beyond CloudWatch metrics to any Firehose-driven streaming-delivery scenario where the destination must be VPC-internal:

  • Application logs routed via Firehose to a VPC-internal log ingestion service.
  • CDC / DynamoDB-Streams output delivered to a VPC-internal event consumer.
  • Application telemetry routed to a self-hosted SIEM or observability platform inside the VPC.

Adjacent patterns

  • AWS PrivateLink + public endpoint as alternative: rather than bridging via Lambda, expose a PrivateLink-based public endpoint. Higher static cost, lower per-record cost, more complex IAM. The Lambda-bridge pattern wins on simplicity at low-to-moderate volumes.
  • NLB with public IP + IP allowlist: terminate Firehose at a public NLB and ACL the IP ranges. Loses the data-stays-in-VPC property; use only when data-privacy isn't load-bearing.
  • Self-hosted aggregator outside VPC + private peering inbound: push Firehose to a public aggregator (3rd-party SaaS or customer-owned), then private-peer the aggregator's egress back in. More moving parts, more attack surface.

Hard problems

  • Cold-start latency on the critical path. Synchronous Firehose transform invocation means cold-start latency back-pressures Firehose's buffer. Provisioned concurrency or warm-keepalive helps.
  • Two-tier retry semantics. Firehose retries the transform invocation; the transform retries the HTTP push. Tuning these two retry budgets in concert is non-trivial; over-retrying amplifies record duplication, under-retrying drops records to S3 fallback.
  • Lambda concurrency caps. A high-volume Firehose stream can saturate the per-account Lambda concurrent-execution quota; sizing those quotas around Firehose buffer + flush cadence is operational discipline.
  • Format translation on the bridge. If the source format (e.g. CloudWatch Streams JSON) differs from the destination format (e.g. OTLP), the Lambda becomes a translator too — expanding what was a transport-only bridge into a format- translation surface.

Seen in

  • sources/2026-05-13-aws-streaming-cloudwatch-metrics-to-vpc-based-opentelemetry-collectors-using-lambda — first canonical wiki home. The customer architecture uses Lambda transform on a Firehose stream that pushes CloudWatch Metric Streams into the customer's VPC-internal OpenTelemetry collector. The post names the structural motivation explicitly ("these endpoints must be public — they cannot be private endpoints inside a VPC"), names the synchronous-invocation contract ("buffers incoming data before synchronously invoking the Lambda function"), and specifies the customer's data-privacy driver ("the metric data and the OpenTelemetry collector to be within their VPC").
Last updated · 542 distilled / 1,571 read