CONCEPT Cited by 1 source
Agent-driven query fan-out¶
Agent-driven query fan-out is the workload shape where the consumer of an analytical query substrate is an AI agent that fires hundreds of parallel queries simultaneously — comparing time windows, validating patterns, exploring hypotheses — instead of the human-paced "write one query, read result, write next query" pattern that analytical substrates were historically optimised for. First wiki canonicalisation 2026-05-27.
Definition¶
A workload satisfies the agent-driven query fan-out shape when:
- The consumer is automated — an AI agent or programmatic process, not a human analyst typing queries one at a time.
- Fan-out is high. Single agent decisions or planning steps fan out into dozens-to-hundreds of independent queries that can run in parallel.
- Latency tolerance is per-query, not per-decision. The agent waits for all parallel queries to complete (or for a timeout) before deciding; tail latency dominates user-perceived latency.
- Data freshness matters. "Agents need data that arrived seconds ago to make good decisions — not a pipeline snapshot from several minutes ago." The minutes-of-pipeline-lag warehouse model breaks the agent's decision quality.
- Queries are not predefined. The agent generates queries based on the current investigation state; the substrate cannot pre-materialise the answers.
Verbatim disclosure (Redpanda 2026-05-27)¶
"A human analyst writes one query, reads the result, writes another. An AI agent fans out across dozens of tables and writes hundreds of queries simultaneously: comparing time windows, validating patterns, and exploring hypotheses in parallel. Agents need data that arrived seconds ago to make good decisions — not a pipeline snapshot from several minutes ago that may no longer reflect what's actually happening."
"Redpanda SQL gives agents standard SQL access to live event streams and full Iceberg history in a single interface, at the freshness and scale their workload actually demands, with a minimal infrastructure footprint, so you can support swarms of those chatty, curious, data-devouring agents."
(Source: sources/2026-05-27-redpanda-redpanda-sql-is-ga-the-query-engine-that-skips-the-pipeline)
Substrate properties required¶
Agent-driven query fan-out makes structural demands on the analytical substrate that human-paced workloads do not:
| Property | Human analyst | AI agent |
|---|---|---|
| Concurrency | Single query in flight | Hundreds of queries in flight |
| Freshness | Minutes of pipeline lag is acceptable | Seconds-of-arrival required |
| Latency budget | Seconds-to-tens-of-seconds per query | Tens-to-hundreds of milliseconds per query |
| Predefined queries | Often via dashboards / saved queries | Largely impossible — generated per investigation step |
| Cost model | Per-analyst per-day | Per-agent per-step × steps × parallelism |
| Tail-latency sensitivity | Low (one user) | High (slow query blocks the agent's decision) |
The combination "freshness + scale + minimal infrastructure footprint" is the load-bearing demand. Warehouse-side ingestion architectures cannot deliver freshness at scale without over-provisioning the ingest path; predefined-query streaming substrates (ksqlDB, Flink SQL) cannot deliver ad-hoc fan-out; external query SaaS cannot deliver minimal infrastructure footprint when agent fan-out turns into thousands of $/query billing events.
Why this differs from human-pace analytics structurally¶
Three structural differences:
- Serial → parallel execution profile. A human analyst's queries are interleaved with thinking time, reading time, and typing time, giving the substrate breathing room between queries. Agents have no breathing room: queries fire as fast as the LLM can generate them, often all at once when the agent plans a step.
- Fewer dashboards, more raw data. Human analysts often work from pre-built BI dashboards over pre-aggregated data. Agents read raw event streams to do their own aggregation in their reasoning loop. The pre-aggregated dashboard substrate doesn't help when the agent doesn't know which aggregation it needs until it's reasoning through the problem.
- No 'wait for the pipeline' tolerance. When a human analyst asks a question and the data is missing because the pipeline is behind, they shrug and check tomorrow. When an agent asks a question and the data is missing because the pipeline is behind, the agent makes a wrong decision based on stale data.
Substrate fit¶
Substrates that fit agent-driven query fan-out:
- In-cluster streaming SQL (concepts/in-cluster-streaming-sql / systems/redpanda-sql) — freshness in milliseconds, Postgres-wire-protocol gives agents standard SQL access, in-VPC data locality avoids per-query egress costs.
- HTAP databases (transactional + analytical on one engine) — freshness from the same write path; bounded by the transactional engine's analytical-query performance.
- MPP query engines on Iceberg / open-table-format substrates (systems/oxla, Genie, etc.) — high parallelism, predicate pushdown, but historically deployed externally to the streaming substrate.
Substrates that don't fit:
- Warehouse-with-ingestion-pipeline — minutes of lag breaks the freshness requirement; per-query billing breaks the agent-fan-out economics.
- Predefined-query streaming SQL ( ksqlDB / Flink SQL) — agents cannot pre-declare their questions.
- Manual ETL + Jupyter notebook — the article's "Option B" narrative; works for one human, breaks for an agent's parallel fan-out.
Connection to agent governance¶
Agent-driven query fan-out at scale interacts with governed agent data access: the substrate must enforce per-agent ABAC / RBAC / audit / on-behalf- of-user authorization on each of the hundreds of parallel queries. Unauthenticated raw-SQL fan-out from agents would be a security disaster. The Redpanda Agentic Data Plane (2025-10- 28) framing positions Oxla / Redpanda SQL inside an authentication + authorization + audit envelope to make the fan-out workload governable.
Canonical wiki instance¶
- systems/redpanda-sql (2026-05-27 GA) — first canonical wiki disclosure of agent-driven query fan-out as the headline motivating workload for an analytical substrate's design. Redpanda's positioning argument: streaming + in-cluster SQL + Postgres protocol + Iceberg history = freshness × ad-hoc-SQL × parallelism × minimal-footprint, the four properties agent fan- out requires together.
Seen in¶
- 2026-05-27 — sources/2026-05-27-redpanda-redpanda-sql-is-ga-the-query-engine-that-skips-the-pipeline — first wiki canonicalisation.
- 2025-10-28 — sources/2025-10-28-redpanda-introducing-the-agentic-data-plane — earlier framing of Oxla as "low-latency context management for agents looking to merge streams or large data sets, search, or simply filter aggregations in real time".
Caveats¶
- Marketing-positioning origin. The fan-out narrative is part of Redpanda's GA-launch positioning vs warehouse-with-ingestion vendors. Real-world per-agent QPS, per-step fan-out distribution, and tail-latency budgets are not disclosed.
- No quantitative scale. "Hundreds of queries simultaneously" is qualitative; benchmarks against agent-frontline workloads not published.
- Cost economics unaddressed. Per-query consumption-based billing × hundred-fold fan-out × thousands of agents may be the cost problem the substrate creates rather than the cost benefit; the post doesn't quantify.
- Agent governance scope. ABAC / row-filtering / column-masking for agent fan-out at GA is positioned by adjacent ADP framing (2025-10-28) but not disclosed concretely in the 2026-05-27 SQL GA post.