CONCEPT Cited by 1 source
Fan-in ingestion¶
Definition¶
Fan-in ingestion is the workload pattern where many independent producers (IoT sensors, application instances, microservices, log shippers) converge their data streams onto a single logical destination (table, topic, bucket). The system's aggregate throughput scales with the number of concurrent producers, not the throughput capacity of any single producer.
Why it matters for system design¶
Fan-in is the dominant production pattern for telemetry, event streaming, and time-series ingestion. Two consequences for architecture:
- Single-client benchmarks are misleading. A powerful host saturates its own bandwidth or CPU before pressuring the ingestion service — it benchmarks the producer, not the system.
- The correct stress model requires coordinated concurrent connections. Tools like Locust on Kubernetes (many pods, each opening independent streams) emulate real-world fan-in more faithfully than single-host load generators.
Structural properties¶
- Throughput scales horizontally with stream/connection count.
- Individual producer failures are isolated — one dropped connection doesn't affect others.
- Back-pressure must be per-stream (not global) to avoid head-of-line blocking.
- The ingestion service must autoscale its backend to match fluctuating producer count.
Seen in¶
- systems/zerobus-ingest — 2,048 concurrent Locust workers (each a K8s pod with one gRPC stream) sustaining 12 GB/s aggregate to a single table. A single host cannot stress-test the service (Source: sources/2026-06-11-databricks-ingesting-the-milky-way-petabyte-scale-with-zerobus-ingest)
Related¶
- systems/kafka — classic fan-in target (many producers → partitioned topic)
- patterns/stream-connection-as-ordering-unit — enables elastic fan-in without static partition planning
- concepts/backpressure — per-stream back-pressure is critical in fan-in systems