Skip to content

CONCEPT Cited by 1 source

Edge inference

Definition

Running ML/AI model inference on hardware physically located at or near the point of use, rather than in a centralized cloud data center. In the generative AI context, this means serving LLM inference on edge devices with constrained GPU memory (typically 16-96 GB VRAM), requiring deliberate trade-offs between model capability, hardware constraints, and operational complexity.

Hardware-driven deployment strategies

When deploying models to multi-GPU edge devices, two fundamental strategies exist:

  1. Model Replication — a full model copy on each GPU, supporting N concurrent requests without queuing. Maximizes throughput at the cost of per-GPU memory consumed by weights (e.g., 81% for a 13 GB model on a 16 GiB T4).

  2. Tensor Parallelism — the model is sharded across all GPUs, leaving remaining memory for KV cache. Supports full context windows for longer interactions but limits concurrent request capacity.

The choice is workload-profile-driven: replication suits high-concurrency short-query environments; tensor parallelism suits fewer concurrent users with complex, long-form interactions.

Key constraints

  • VRAM ceiling: Model must fit within device memory after quantization (GGUF/GPTQ/AWQ formats)
  • No elastic scaling: Fixed hardware means capacity planning is static
  • Thermal/power: Industrial environments may have power/cooling constraints
  • Update velocity: Models updated via deployment cycles, not hot-swapped

Seen in

Last updated · 590 distilled / 1,788 read