Skip to content

AWS 2026-08-13

Read original ↗

Adobe Firefly: Simplified observability with Amazon Managed Prometheus

Summary

Adobe Firefly moved the critical metrics tier of its GPU-training observability stack from self-managed Prometheus toward Amazon Managed Service for Prometheus (AMP). The production training substrate is EKS across thousands of compute nodes and GPUs, where job-, pod-, node-, and GPU-health telemetry has high cardinality and queries can cross more than one billion data points. Instead of a disruptive replacement, Adobe ran AMP managed scrapers beside the legacy Prometheus deployment and migrated the critical metric tier first. The result is a managed, horizontally scalable time-series backend with materially faster GPU queries and a longer practical diagnostic window. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)

Key takeaways

  1. GPU training makes telemetry multidimensional. A useful diagnosis combines GPU utilisation, memory, network throughput, pod status, node readiness, resource allocation, and GPU health; one aggregate utilisation measure cannot explain a distributed-training bottleneck. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)
  2. Metric cardinality, not just sample rate, defines the query challenge. The article's example of 2,000 nodes and 16,000 GPUs scraped every 30 seconds can produce more than one billion data points in a query window. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)
  3. Migrate the critical tier independently. Adobe moved 2 million critical time series to AMP while retaining its self-managed Prometheus deployment, limiting operational disruption and preserving established monitoring workflows. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)
  4. Parallel collection is the migration seam. AMP managed scrapers collect from the EKS clusters and forward directly to AMP workspaces while the existing Prometheus deployment remains in place; the source describes this as an incremental adoption path, not a big-bang replacement. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)
  5. The managed backend changes the query envelope. Reported GPU-utilisation query speedups were 3.5× for four-hour windows, 22.6× for 12-hour windows, and 28.8× for 24-hour windows. Queries that had timed out at 60 seconds or returned partial results after two minutes complete in about 10 seconds after the migration. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)
  6. Longer windows improve debugging, not only dashboards. The practical visibility limit rose from six to 24 hours, allowing engineers to correlate a 256+-node training job's lifecycle with infrastructure events and identify when a degradation began. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)
  7. Managed service limits and pricing remain design inputs. The cited AMP workspace limit is up to 50 million active time series, with stated headroom to one billion as infrastructure grows; AMP and AMG charges are based on metrics ingested, stored, and queried. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)

Architecture

EKS GPU training clusters
  ├── job / GPU / pod / node / network metrics
  ├── self-managed Prometheus ── existing workflow retained
  └── AMP managed scrapers ────> AMP workspace ────> Amazon Managed Grafana
                                      │
                                      └── high-cardinality PromQL queries

The article only establishes parallel metric scraping during the transition. It does not document whether both stores receive every series, how dashboard and alert cutover is sequenced, or the eventual retention policy for the legacy Prometheus stack. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)

Operational evidence

Measure Before / self-managed Prometheus After / AMP Reported effect
Critical metric volume N/A 2 million time series migrated Critical tier moved first
Example fleet 2,000 nodes; 16,000 GPUs Same workload shape >1 billion data points per query window at 30-second scrape interval
4-hour GPU query Baseline AMP 3.5× faster
12-hour GPU query Baseline AMP 22.6× faster
24-hour GPU query Baseline AMP 28.8× faster
Practical investigation window 6 hours 24 hours Full long-running-job lifecycle visible
Problematic query completion 60-second timeout or partial result after 2 minutes ~10 seconds >28× improvement claimed
Workspace capacity N/A Up to 50M active time series Stated service limit; not Adobe's measured total capacity

Systems, concepts, and patterns extracted

Caveats

  • This is an AWS/customer case study, not an independent benchmark; query definitions, hardware, PromQL expressions, and percentile distributions are not provided.
  • The 50-million-series workspace figure is a service limit, not evidence that Adobe runs at that total.
  • No end-to-end cost comparison, ingestion rate, retention period, alerting cutover plan, or availability SLO is published.
  • The source says Adobe intends to extend AMP to remaining metric tiers and a multi-tenant highly available stack, so the described deployment is an intermediate state.
  • The article reports query performance primarily for GPU-utilisation metrics; it says the result was consistent for memory, power, and thermal metrics but provides no per-metric timings.

Source

Last updated · 622 distilled / 1,953 read