SYSTEM Cited by 1 source
Adobe Firefly training observability¶
Adobe Firefly's training observability system is an EKS-based GPU telemetry architecture for training workloads spread over thousands of compute nodes and GPUs. Its documented evolution retains self-managed Prometheus while AMP managed scrapers forward a critical metric tier to Amazon Managed Service for Prometheus. The architecture makes high-cardinality job, GPU, Kubernetes, and hardware-health telemetry queryable over a longer diagnostic horizon. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)
Documented shape¶
- EKS hosts the GPU-training clusters.
- Self-managed Prometheus remains available during the transition.
- AMP managed scrapers collect the selected metrics and forward them to AMP workspaces.
- Amazon Managed Grafana is the stated native dashboard integration.
- Engineers use job-level GPU utilisation, memory, and network throughput; pod and node health; and GPU-health signals to find distributed-training bottlenecks.
Observed scale and outcome¶
The source's representative query workload covers 2,000 nodes and 16,000 GPUs scraped every 30 seconds, exceeding one billion data points in a query window. Adobe had moved 2 million critical time series to AMP. Reported 24-hour GPU-utilisation queries were 28.8× faster, and the practical diagnostic window grew from six to 24 hours. (Source: sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus)
Caveats¶
The source does not disclose its EKS cluster count, full ingestion rate, retention configuration, dashboard design, alerting topology, cost delta, or a final date for retiring self-managed Prometheus.
Seen in¶
- sources/2026-08-13-aws-adobe-firefly-simplified-observability-with-amazon-managed-prometheus — the only current source for this customer architecture.