Skip to content

DATABRICKS 2026-07-06

Read original ↗

Scaling Security Alert Triage With Specialized Agents on Databricks

Summary

Databricks' Incident Response team built a fleet of 17 source-specific triage agents to automatically review 100% of low-severity security alerts. Rather than using a single generalist LLM to decide what to escalate (which produced a 50% false escalation rate), they decomposed the problem into per-detection-source agents, each with its own deterministic filters, enrichment pipeline, specialized prompts, and cost controls. Agents run on Spark Structured Streaming, persist decisions to Delta tables, trace everything via MLflow, and are evaluated against analyst ground-truth datasets.

Key takeaways

  1. Single-agent approach failed at 50% escalation rate — a generalist model lacked the per-source behavioral context needed to distinguish noise from signal; splitting into 17 source-specific agents dramatically reduced false positives.

  2. Deterministic filtering handles 30–95% of alert volume without any LLM call — programmatic rules check for known-benign patterns (trusted IP lists, expected service-account activity) and produce instant dispositions. "The cheapest call is the one you never make."

  3. Dedicated Threat Intelligence (TI) agent as shared service — any triage agent can call the TI agent to enrich an indicator (IP, domain, hash) with structured assessment (known malicious / unknown / benign) from authoritative threat-intel sources.

  4. Prompt functions map alert titles to domain-specific instructions — each detection type gets a specialized prompt function tuned to that source's false-positive patterns and behavioral baselines.

  5. Three-layer cost management — (a) deterministic filtering eliminates the cheapest volume, (b) per-batch configurable spend cap stops processing if exceeded, (c) daily alert cap provides a hard ceiling regardless of inbound volume; per-category tool-call budgets prevent runaway discovery loops.

  6. Structured Streaming for real-time ingestion and routing — alerts are ingested via Spark Structured Streaming, enriched, and routed to the appropriate sub-agent.

  7. MLflow Tracing captures every agent decision end-to-end — inputs, intermediate steps, and final outputs are recorded; analyst labels on traces build the ground-truth benchmark for evaluating prompt changes.

  8. Agent-escalated alerts were ~10× more likely to be true positives than existing HIGH/MEDIUM severity alerts — demonstrating that context-rich agent triage outperforms severity-based prioritization.

  9. LLMs hallucinate on high-entropy security data — hashes, random subdomains, and generated filenames are deliberately pattern-free and defeat language models. Solution: use the model for reasoning, not recall; retrieve specific artifact values via tool calls to authoritative sources.

  10. Context is the biggest performance lever — adding 6 months of alert history per user, per-alert-type false-positive rates, and explicit behavioral patterns to prompts yielded the largest quality gains.

Operational numbers

Metric Value
Alerts triaged 18,000+
Escalation rate 3.2%
Median triage time 10.5 seconds
Analyst hours saved (first 30 days) 6,500+
Source-specific agents 17
Deterministic filter coverage 30–95% of volume per source
Escalation quality vs. HIGH/MEDIUM ~10× more likely true positive
One source FP reduction 72% → 3.4%
Suspicious domains identified 22

Architecture

Alerts → Structured Streaming → [TI Agent enrichment] → Source router
                                                            │
                    ┌───────────────────────────────────────┤
                    ▼                                       ▼
            Agent-1 (Source A)                      Agent-17 (Source Q)
            ┌─────────────────┐                    ┌─────────────────┐
            │ Deterministic   │                    │ Deterministic   │
            │ filter (30-95%) │                    │ filter          │
            │       ↓         │                    │       ↓         │
            │ Context enrich  │                    │ Context enrich  │
            │       ↓         │                    │       ↓         │
            │ Prompt function │                    │ Prompt function │
            │       ↓         │                    │       ↓         │
            │ LLM reasoning   │                    │ LLM reasoning   │
            │       ↓         │                    │       ↓         │
            │ Disposition:    │                    │ Disposition:     │
            │ escalate/       │                    │ escalate/        │
            │ monitor/close   │                    │ monitor/close    │
            └────────┬────────┘                    └────────┬────────┘
                     └──────────────────┬───────────────────┘
                                        ▼
                              Delta table (decisions)
                              MLflow trace (full audit)
                                        │
                                        ▼
                              IR queue (escalations only)
                              Analyst labels → ground truth

Caveats

  • The system is described only for low-severity alerts — the team still manually triages HIGH/MEDIUM.
  • Evaluation beyond false-positive rate (missed escalations) depends on the Databricks Review App, which is described as planned, not yet deployed at publication time.
  • Cost controls imply that under extreme volume spikes, some alerts are skipped (recorded but not processed), which is an accepted trade-off.

Source

Last updated · 585 distilled / 1,765 read