Skip to content

AWS

Read original ↗

Architecting offline-first generative AI applications for edge deployments using AWS services

Summary

AWS Architecture Blog reference architecture for deploying generative AI at the edge — moving AI inference to locations where cloud connectivity is unreliable or unavailable. The architecture addresses industrial environments (manufacturing, offshore platforms, remote agriculture) where operators need instant access to equipment documentation, log correlation, and troubleshooting guidance without depending on cloud connectivity. The solution spans three layers: cloud-side model preparation (Amazon Bedrock + SageMaker AI Pipelines for FMOps), deployment orchestration (AWS IoT Greengrass), and edge-side inference (Ollama + Strands Agents + ChromaDB RAG).

Key takeaways

  1. Offline-first as an architectural pattern: The architecture prioritizes local inference capability over cloud connectivity, treating the cloud as a model-customization and continuous-improvement layer rather than a runtime dependency. Edge operations function independently during connectivity gaps.

  2. Four model customization strategies ranked by complexity: Fine-tuning (FT) for task-specific behavior, continued pre-training (CPT) for domain-knowledge injection, hybrid FT+RAG for balanced capability/complexity, and full CPT+RAG+FT for maximum quality — the post recommends hybrid FT+RAG as the right balance.

  3. Hardware-constrained model deployment creates a structural choice: With gpt-oss-20b on 4× NVIDIA T4 GPUs (16 GiB each), two deployment strategies are available: model replication (full copy per GPU, 4 concurrent requests, 81% VRAM consumed by weights) vs tensor parallelism (model sharded across GPUs at 3.2 GB/GPU, leaving ~12.8 GB for KV cache enabling full 128K context). Workload profile determines choice: replication for high-concurrency short queries, tensor parallelism for fewer users with longer interactions.

  4. RAG pipeline operates entirely on CPU/SSD: ChromaDB with SQLite + HNSW defaults, sentence-transformer embedding model (384 dimensions) on CPU. Documents chunked at 512 tokens with 50-token overlap. Dataset up to 5 GB yields sub-50ms retrieval latency. Zero GPU VRAM consumption, keeping full 16 GB dedicated to the LLM.

  5. FMOps pipeline pattern: Amazon Nova Pro on Bedrock generates structured Q&A pairs from raw documentation → SageMaker AI Pipelines orchestrates automated fine-tuning of gpt-oss-20b (21B total params, 3.6B active per token, MoE architecture, 32 experts, 128K context) → quantized model artifact stored in S3 → IoT Greengrass deploys to edge device.

  6. Strands Agents as edge orchestration layer: Routes requests to appropriate tools (RAG knowledge base, device telemetry). Agent-based pattern provides extensibility — new data sources added without modifying the core inference pipeline.

  7. Asynchronous feedback loop for continuous improvement: Usage data queued locally during offline operation, synchronized opportunistically when connectivity is available. Creates a virtuous cycle: edge usage → cloud model refinement → Greengrass redeploys improved model back to device.

  8. Security perimeter shifts to the edge: At the edge you own the full stack — physical access, encryption at rest (LUKS/BitLocker + TPM), encryption in transit (TLS 1.2+ even on loopback), network segmentation (operator portal / inference runtime / cloud sync in separate zones), prompt guardrails for injection protection, and IAM least-privilege per AWS service in the cloud pipeline.

  9. Evaluation results validate hybrid approach: Fine-tuned gpt-oss-20b + RAG vs base + RAG scored by three LLM judges (Claude 4.5 Haiku, Claude 4.5 Sonnet, Amazon Nova Pro) on a 12-point scale: FT+RAG consistently outperforms base+RAG by 10-17 percentage points (85% vs 68.3% on Haiku, 76.7% vs 61.7% on Sonnet, 82.5% vs 72.5% on Nova Pro).

Operational numbers

Metric Value
Model gpt-oss-20b (MoE: 21B total, 3.6B active/token, 32 experts, 128K context)
Edge hardware (reference) g4dn.12xlarge: 4× NVIDIA T4 (16 GiB each), 48 vCPUs, 129 GiB RAM
Model memory per GPU (replicated) 13 GB / T4 (81% of 16 GiB)
Model memory per GPU (tensor parallel) 3.2 GB / T4 (leaving ~12.8 GB for KV cache)
RAG embedding dimensions 384
RAG chunk size 512 tokens, 50-token overlap
RAG dataset limit Up to 5 GB
RAG retrieval latency <50 ms
Evaluation improvement (FT+RAG vs base+RAG) +10–17 percentage points across 3 judges
Downtime cost (Fortune 500, Siemens 2024) $1.4 trillion annually

Caveats

  • Reference architecture, not a production retrospective — no real deployment metrics (latency, throughput, cost, uptime) disclosed.
  • Hardware examples span two devices inconsistently (g4dn.12xlarge with 4×T4 in one section, Jetson Xavier with 16+ GB in another) without reconciling.
  • No discussion of model update frequency, edge storage lifecycle, or version rollback mechanics.
  • No cost analysis comparing edge inference to cloud inference with connectivity buffering.
  • Positioned as generalizable but only one use case (manufacturing equipment documentation) is demonstrated.

Source

Last updated · 590 distilled / 1,788 read