SLACK 2025-12-01

Slack — Streamlining security investigations with agents¶

Summary¶

Slack's Security Engineering team built an internal multi-agent system (Spear, announced in this post as the first in a series) that triages security-detection alerts during on-call shifts. The prototype — a 300-word prompt plus an MCP server exposing data sources — produced wildly variable quality: "sometimes it would produce excellent, insightful results [...] however, sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods." The core design lesson is verbatim: "prompts are just guidelines; they're not an effective method for achieving fine-grained control." Slack's production design replaces the single-prompt investigation with a three-persona agent team (Director / Expert / Critic) plus per-task structured output, three investigation phases (discovery → trace → conclude), and a knowledge pyramid that tiers model cost against task cognitive load (cheap experts, mid-tier critic, expensive director). The service runs as a three-component stack — Hub (API + storage + metrics), Worker (pulls queued investigations), Dashboard (real-time observation + replay) — with investigation events streamed from worker back through hub to dashboard. Post discloses an edited worked example where the Critic noticed a credential exposure the Expert missed; the Director then pivoted the investigation and surfaced both the security issue and the expert's analytical blind spot in the final report. First post in a series; future posts promised on alignment/orientation, artifacts as inter-agent channel, and human/agent collaboration.

Key takeaways¶

Prompts are guidelines, not control. Slack's verbatim lesson after trying to harden the single-prompt prototype by stressing "question assumptions, verify data from multiple sources, make use of the complete set of data sources": "While we did have some success with this approach, ultimately prompts are just guidelines; they're not an effective method for achieving fine-grained control." The architectural consequence is that every piece of investigation guidance that matters must become its own model invocation with its own structured-output schema, not a bullet in a system prompt. (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Decomposition into per-task model invocations is Slack's answer. "Our solution was to break down the complex investigation process we'd described in the prompt of our prototype into a sequence of model invocations, each with a single, well-defined purpose and output structure. These simple tasks are chained together by our application." The application, not the prompt, owns sequencing, state, and context propagation — canonical instance of patterns/one-model-invocation-per-task. (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Structured outputs have costs but deliver control. "Using structured outputs isn't 'free'; if the output format is too complicated for the model, the execution can fail. Structured outputs are also subject to the usual problems of cheating and hallucination." Slack ships anyway because "this approach gave us more precise control at each step of the investigation process" and because each per-task schema is small enough to stay within the model's competence. Load- bearing instance of concepts/structured-output-reliability applied to security investigation: a malformed output is not a cosmetic issue, it breaks the orchestration graph that the application relies on. (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Three-persona team (Director / Expert / Critic). Slack's verbatim role split — Director: "the responsibility is to progress the investigation from start to finish. The Director interrogates the experts by forming a question [...] The Director uses a journaling tool for planning and organizing the investigation as it progresses." Expert (four of them in prod: Access, Cloud, Code, Threat): "Each domain expert has a unique set of domain knowledge and data sources. The experts' responsibility is to produce findings from their data sources in response to the Director's questions." Critic: "a 'meta-expert.' The Critic's responsibility is to assess and quantify the quality of findings made by domain experts using a rubric we've defined. The Critic annotates the experts' findings with its own analysis and a credibility score for each finding." Canonical instance of patterns/director-expert-critic-investigation-loop. (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Critic as weakly-adversarial control on hallucination. Verbatim: "The weakly adversarial relationship between the Critic and the expert group helps to mitigate against hallucinations and variability in the interpretation of evidence." The Critic is pointed at the experts, not cooperating with them, but is not fully adversarial — it shares the investigation's goal but audits the method. Canonical instance of concepts/weakly-adversarial-critic. (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Knowledge pyramid tiers model cost against task. Verbatim framing of the cost/value gradient: "At the bottom of the knowledge pyramid, domain experts generate investigation findings by interrogating complex data sources, requiring many tool calls. Analyzing the returned data can be very token-intensive. Next, the Critic's review identifies the most interesting findings from that set [...] it assembles an up to date investigation timeline, integrating the running investigation timeline and newly gathered findings into a coherent narrative. The condensed timeline, consisting only of the most credible findings, is then passed back to the Director. This design allows us to strategically use low, medium, and high-cost models for the expert, critic, and director functions, respectively." Canonical instance of concepts/knowledge-pyramid-model-tiering — cheap models at the fan-out layer where tool calls dominate, mid-tier in the reviewer layer where reasoning-density is higher, most expensive model at the apex where the task is strategic planning over a condensed summary. (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Three investigation phases. Discovery: "the goal [...] is to ensure that every available data source is examined. The Director reviews the state of the investigation and generates a question that is broadcast to the entire expert team." Director Decision: a "meta-phase in which the Director decides whether to advance to the next investigation phase or continue in the current one." Trace: "Once the discovery phase has made clear which experts are able to produce relevant findings, the Director transitions the investigation to the trace phase. In the trace phase, the Director chooses a specific expert to question. We also have the flexibility to vary the model invocation parameters by phase, allowing us to use a different model or enhanced token budget." Conclude: "when sufficient information has been gathered to produce the final report." Canonical instance of patterns/phase-gated-investigation-progression + concepts/investigation-phase-progression. The Director is responsible for advancing the phase; phase is carried as application state, not in the prompt. Phases also serve as per-phase knobs for model/budget selection — "flexibility to vary the model invocation parameters by phase". (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Hub / Worker / Dashboard service architecture. Verbatim: "Our prototype used a coding agent CLI as an execution harness, but that wasn't suitable for a practical implementation." Landing design: "Hub — provides the service API and an interface to persistent storage. Besides the usual CRUD-like API, the hub also provides a metrics endpoint so we can visualise system activity, token usage, and manage cost. Worker — Investigation workers pick up queued investigation tasks from the API. Investigations produce an event stream which is streamed back to the hub through the API. Workers can be scaled to increase throughput as needed. Dashboard — used by staff to interact with the service. Running investigations can be observed in real-time, consuming the event stream from the hub. Additionally the dashboard provides management tools, letting us view the details of each model invocation. This capability is invaluable when debugging the system." Canonical instance of patterns/hub-worker-dashboard-agent-service. The coding- agent-CLI-as-harness → hub+worker+dashboard rewrite is the second decisive abandonment of a prototype shortcut in the post (first being the single-prompt design). (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Emergent behaviour: Critic catches what Expert misses. Slack's edited worked example: the detection system raised an alert for a specific command sequence ("can be an indicator of compromise"). The Expert investigated the process ancestry and "incorrectly assessed credential handling as secure". The Critic — auditing the Expert's findings against the rubric — identified that a credential had been exposed in process command-line parameters within the ancestry chain. The Director, on receiving the Critic's annotation, pivoted the investigation to focus on the credential exposure. The final report surfaced both the security finding and the Expert's "analysis blind spots that require attention." Verbatim: "What is notable about this result is that the expert did not raise the credential exposure in its findings; the Critic noticed it as part of its meta-analysis of the expert's work. The Director then chose to pivot the investigation to focus on this issue instead." The credential exposure was referred to the owning team for remediation. (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)
Supervision rather than collection. Slack's on-call shift mode shifts from "the laborious work of gathering evidence" to "supervising investigation teams." Unlike static detection rules, the agents "often make spontaneous and unprompted discoveries." (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)

Systems / concepts / patterns extracted¶

Systems - systems/slack-spear (new) — Slack's unnamed-in-post-but- internally-Spear security-investigation agent service (name inferred from the post's spear-* image slug convention; if the product name is disclosed elsewhere, the page should be updated). Three-component service: Hub (API + storage + metrics) / Worker (pulls investigations, streams events back) / Dashboard (real-time observe + replay + per-invocation debugging). Executes the Director/Expert/Critic investigation loop across discovery / trace / conclude phases. Built on MCP stdio servers for data-source access during prototype; production harness not fully disclosed. - systems/model-context-protocol (extend) — stdio-mode MCP server used to "safely expose a subset of our data sources through the tool call interface" during Spear's prototype. First wiki instance of MCP in an internal security- investigation pipeline (prior MCP coverage: dev tooling, public MCP registries, Cloudflare's MCP ecosystem).

Concepts (new) - concepts/structured-output-reliability (extend) — Slack's verbatim framing that structured outputs "isn't 'free'", subject to "cheating and hallucination", but deliver per-step "precise control". New Seen-in canonicalises structured output as orchestration-boundary discipline in a multi-agent investigation loop. - concepts/knowledge-pyramid-model-tiering (new) — the three-tier model-cost-vs-task-cognitive-load gradient: cheap/high-fan-out at the leaves (Experts tool-calling over data sources), mid-tier in the middle (Critic reading the leaves' work + assembling timeline), expensive at the apex (Director planning + deciding investigation progression). - concepts/investigation-phase-progression (new) — the progression of an investigation through named phases (discovery → trace → conclude), with a meta-phase for phase-advancement decisions. Phases are application state, not prompt state; each phase has its own model-parameter envelope. - concepts/weakly-adversarial-critic (new) — the architectural stance where a critic agent audits peer agents' work for hallucination / sloppiness / methodological gaps without being fully adversarial (it shares the task's goal). Contrast with pure adversarial (red-team) or pure cooperative (helper) critic stances. - concepts/prompt-is-not-control (new) — Slack's verbatim lesson: "prompts are just guidelines; they're not an effective method for achieving fine-grained control." Architectural consequence: fine-grained control lives in application code (per-task invocations, schemas, state machines), not prompts.

Patterns (new) - patterns/director-expert-critic-investigation-loop — the three-persona agent team (Director = planner, N Experts = domain tool-users, Critic = meta-reviewer) with a loop shape of question → expert findings → critic review → director progresses investigation. Canonical instance: Slack Spear. - patterns/one-model-invocation-per-task — decompose a complex agent prompt into a sequence of separate model invocations, each with a single well-defined purpose and a task-specific structured-output schema. The application, not the prompt, owns sequencing, state, and context propagation. - patterns/hub-worker-dashboard-agent-service — productize an agent loop as three components: Hub (API + storage + metrics), Worker (queue consumer + event emitter), Dashboard (real-time event consumer + per-invocation debugger). Events flow worker → hub → dashboard; investigations can be observed live or replayed from persisted state. - patterns/phase-gated-investigation-progression — gate an agent loop's behaviour on explicit named phases (discovery / trace / conclude + a Director-Decision meta-phase). Phase-transition decisions are first-class, and each phase can have different model-tier / token-budget / tool-surface parameters.

Patterns (extended) - patterns/specialized-agent-decomposition — Slack's four- Expert split (Access / Cloud / Code / Threat) is canonical security-domain instance at the peer-agent altitude; pairs with the Director/Critic supra-agent structure. - patterns/multi-round-critic-quality-gate — Slack's Critic is a single-round critic (not a multi-round fixer loop as in Meta's tribal-knowledge pipeline) but shares the independent- critic-reduces-hallucination rationale. New Seen-in canonicalising the live-investigation altitude variant. - patterns/drafter-evaluator-refinement-loop — Slack's Expert-then-Critic shape is the security-investigation altitude variant of Lyft's Drafter-then-Evaluator translation loop. Slack's variant adds the Director-layer deciding what to do with the Critic's output rather than just retrying.

Operational numbers disclosed¶

Prototype vintage: end of May 2025 (~6 months before this December 2025 post).
Prompt size (prototype): 300 words.
Prompt sections (prototype): 5 (Orientation, Manifest, Methodology, Formatting, Classification).
Domain experts in production: 4 (Access, Cloud, Code, Threat).
Investigation phases: 3 (Discovery, Trace, Conclude) + 1 meta-phase (Director Decision).
Service components: 3 (Hub, Worker, Dashboard).

No throughput / latency / cost / token-usage / hallucination- rate / false-positive-rate numbers disclosed. No disclosure of specific model families, model versions, data-source names, or the rubric the Critic uses. No disclosure of failure modes observed in production (beyond the worked example showing Expert blind spot detected).

Caveats¶

First post in a promised series — "There's a great deal more to say. We look forward to sharing more details of how our system works in future blog posts." Future posts promised on: (1) "maintaining alignment and orientation during multi-persona investigations"; (2) "using artifacts as a communication channel between investigation participants"; (3) "human in the loop: human / agent collaboration in security investigations". Ingest is structured to expect follow-up mechanism depth on these axes.
No production numbers on throughput, cost, accuracy, or false-positive reduction. Claims of "meaningful benefits" and "many times" for unprompted discoveries are qualitative.
No model/vendor disclosure — the three tiers of the knowledge pyramid are characterised as "low, medium, and high-cost models" without naming families. Structured-output mechanism is not disclosed (JSON schema? Pydantic? vendor- specific structured-output mode?).
Worked example is edited. "We've included an edited investigation report" — the report summary quoted is acknowledged as "a lightly edited version." The credential-exposure-in-process-ancestry finding is genuine but the exact trigger command and data sources consulted are elided for security reasons.
Rubric is named but not disclosed. "The Critic's responsibility is to assess and quantify the quality of findings made by domain experts using a rubric we've defined." The rubric itself — what dimensions it scores on, how it's calibrated, how disputes are resolved — is out of scope for this post.
"Weakly adversarial" is asserted, not measured. Slack frames the Critic/Expert dynamic as weakly adversarial but doesn't disclose how the Critic is prompted to be oppositional, whether different model families run the two roles, or whether there's any measured reduction in Expert- alone hallucination rate.
MCP stdio mode is named as the prototype substrate but "our prototype used a coding agent CLI as an execution harness, but that wasn't suitable for a practical implementation" — unclear whether MCP persists in the production Hub/Worker architecture or whether the worker speaks directly to data sources.
Spear name is inferred from image-asset URL slug patterns (investigation_round_simple_vertical_ad2b54.png, knowledge_pyramid_retina.png, investigation_phases_vertical_final_300dpi.png, process_ancestry_retina.png) and not explicitly stated in the post body. If a different product name is disclosed in a future post, the system page should be renamed.
Acknowledgements: Chris Smith, Abhi Rathod, Dave Russell, Nate Reeves.
Series: first post in a Slack Security Engineering series.

Cross-source continuity¶

First Slack ingest on security-engineering axis. Prior Slack coverage spans developer-productivity (Enzyme→RTL, Bazel/Quip), reliability-engineering (Deploy Safety), test-framework-integration (Axe + Playwright), mobile- accessibility (Android VPAT), fleet-configuration (Chef phase 2), and backend-architecture (Unified Grid). This post opens the sixth axis: security-engineering / AI-agent operations.
Companion to Cloudflare AI Code Review. Cloudflare's system is the closest public architectural sibling — a coordinator + specialised sub-reviewers + structured-output review at scale — but at the code-review altitude rather than live-security-investigation altitude. Slack's Director/Critic structure adds the meta-reviewer layer that Cloudflare's judge-pass-inside-coordinator consolidates into a single agent. Both canonicalise the coordinator-over-specialists shape with structured output as orchestration boundary.
Extension of patterns/specialized-agent-decomposition at the security-operations altitude — Slack's four experts (Access / Cloud / Code / Threat) are cohesive-scope-toolset agents whose collaboration is coordinated by a supra-agent (Director) rather than peer-to-peer. Different shape from Databricks' storex peer-collaboration framing.
Sibling to patterns/drafter-evaluator-refinement-loop (Lyft localization) and patterns/multi-round-critic-quality-gate (Meta tribal-knowledge) — all three separate a producer from an evaluator on structural grounds. Slack adds the Director-above-Critic third layer that decides what to do with the evaluator's output (progress investigation? pivot? conclude?) rather than just retrying.
Companion to patterns/four-component-agent-production-stack (Redpanda Openclaw) at the productisation altitude — Redpanda's Gateway + Audit + Token-vault + Sandboxed-compute is the enterprise-agent substrate; Slack's Hub + Worker + Dashboard is the single-agent-service productisation shape that sits on top of such a substrate.
Companion to concepts/agentic-ai-infrastructure-challenges (Redpanda Akidau talk recap) — Slack's system exercises context-building (data-source-to-expert routing), auditing (full per-invocation replay in Dashboard), replay + validation (Dashboard lets staff "view the details of each model invocation"), routing (phase-gated Director decisions), and multi-agent coordination (Director/Expert/Critic loop).