SYSTEM Cited by 2 sources

Slack Spear¶

Slack's internal multi-agent security-investigation service that triages detection-system alerts during on-call shifts. Built by Slack's Security Engineering team; first announced in a December 2025 Slack Engineering post as the opening of a series. Name inferred from the post's image-asset URL slugs and internal Slack convention; the post itself uses the generic phrase "our service". If the product's real name differs, this page should be renamed.

What it does¶

Spear ingests security-detection alerts and runs an automated end-to-end investigation: gathers evidence across Slack's security data-source fleet, cross-references findings, evaluates credibility, and produces a structured report with a classification and recommended actions. The on-call engineer supervises running investigations in real time via the Dashboard rather than manually performing evidence collection.

Verbatim: "During our on-call shifts, we're switching to supervising investigation teams, rather than doing the laborious work of gathering evidence." (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)

Agent architecture¶

Three persona categories and a phased investigation loop.

Director agent (1 per investigation)¶

Runs on the highest-cost model tier in the knowledge pyramid (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents).
Forms questions to the Expert team; uses a journaling tool for planning.
Consumes the condensed investigation timeline produced by the Critic; does not see raw Expert tool calls.
Decides investigation phase progression (discovery → trace → conclude) via an explicit Director-Decision meta- phase.
In the trace phase, chooses a single Expert to question rather than broadcasting.

Expert agents (4 in production)¶

Access — authentication, authorization, perimeter.
Cloud — infrastructure, compute, orchestration, network.
Code — source-code and configuration-management analysis.
Threat — threat analysis + intelligence sources.

Each has a unique toolset and domain prompt; all run on low-cost models because leaf-layer work is tool-call-heavy and token-intensive (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents).

Critic agent (1 per investigation)¶

Runs on medium-cost models in the knowledge pyramid.
Audits each Expert finding against a rubric, annotating with analysis + credibility score.
Assembles the investigation timeline from most-credible findings.
Stance: "weakly adversarial" to the Expert group — shares the investigation's goal, but audits the method. See concepts/weakly-adversarial-critic.

Investigation phases¶

Phase	Director action	Who sees the question
Discovery	Broadcast a single question to the whole Expert team	All Experts
Director Decision	Meta-phase: advance or remain	N/A (Director only)
Trace	Pick one specific Expert to question; may vary model parameters / token budget by phase	One Expert
Conclude	Produce the final report	N/A (Director only)

"The Director persona is responsible for advancing the phase." Phase is application state, not prompt state. Per-phase model-parameter envelopes allow the system to "use a different model or enhanced token budget" per phase (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents).

See patterns/phase-gated-investigation-progression.

Knowledge pyramid¶

Slack's explicit model-cost tiering against task cognitive load (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents):

Tier	Agent	Cost	Rationale
Apex	Director	High	Strategic planning over a condensed timeline; few tokens in, few decisions out
Middle	Critic	Medium	Reads tool calls + findings, applies rubric, writes timeline; reasoning-dense
Base	Experts	Low	Many tool calls + large data returns; low per-call cognitive load, high fan-out

See concepts/knowledge-pyramid-model-tiering.

Investigation loop¶

Director (plans) ─► Expert(s) (tool-call data sources, emit findings)
         ▲                                              │
         │                                              ▼
         │                                  Critic (audits, rubric-scores,
         │                                   assembles condensed timeline)
         │                                              │
         └──────────────────────────────────────────────┘
              Director reads timeline,
              decides next question + phase

See patterns/director-expert-critic-investigation-loop.

Canonical single-investigation invocation flow (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents):

Alert arrives; investigation queued to Worker.
Discovery phase: Director broadcasts a question to all four Experts; Experts produce findings; Critic reviews all findings, scores them, assembles a timeline.
Director Decision: Director reads timeline; decides whether to stay in discovery or advance.
Trace phase: Director picks a specific Expert; repeats question/findings/critic cycle with phase-specific model parameters.
Conclude phase: Director produces final report classification + written analysis.

Service architecture¶

Three components (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents):

Hub¶

"Provides the service API and an interface to persistent storage. Besides the usual CRUD-like API, the hub also provides a metrics endpoint so we can visualise system activity, token usage, and manage cost."

Worker¶

"Investigation workers pick up queued investigation tasks from the API. Investigations produce an event stream which is streamed back to the hub through the API. Workers can be scaled to increase throughput as needed."

Dashboard¶

"Used by staff to interact with the service. Running investigations can be observed in real-time, consuming the event stream from the hub. Additionally the dashboard provides management tools, letting us view the details of each model invocation. This capability is invaluable when debugging the system."

See patterns/hub-worker-dashboard-agent-service.

Event flow: Worker → Hub → Dashboard (via hub's event API bridging persistent store to live consumers).

Prototype history¶

End of May 2025 prototype: a 300-word system prompt with five sections (Orientation / Manifest / Methodology / Formatting / Classification), a stdio-mode MCP server exposing a subset of data sources, and a coding-agent CLI repurposed as the execution harness (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents).

Prototype failure modes that drove the rewrite:

Wildly variable quality. "Sometimes it would produce excellent, insightful results with an impressive ability to cross-reference evidence across different data sources. However, sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods."
Prompt-hardening didn't work. "We spent some time trying to refine our prompt, stressing the need to question assumptions, to verify data from multiple sources, and to make use of the complete set of data sources. While we did have some success with this approach, ultimately prompts are just guidelines; they're not an effective method for achieving fine-grained control." See concepts/prompt-is-not-control.
Coding-agent-CLI-as-harness "wasn't suitable for a practical implementation."

Two structural rewrites followed: single-prompt → per-task structured-output invocations; CLI harness → Hub/Worker/ Dashboard service.

Emergent behaviour (canonical worked example)¶

Alert raised for a suspicious command sequence. The investigation's Expert reviewed the process ancestry and "incorrectly assessed credential handling as secure". The Critic, reviewing the Expert's findings against the rubric, identified a credential exposure in a process command- line parameter within the ancestry chain. The Director, on receiving the Critic's annotation, pivoted the investigation to focus on the credential exposure rather than the original suspicious command.

The final report surfaced both findings:

Credential Exposure (primary): the leaked credential was referred to the owning team for remediation.
Expert-Critic Contradiction (secondary): the Expert's blind spot was flagged as "indicating analysis blind spots that require attention."

Verbatim: "What is notable about this result is that the expert did not raise the credential exposure in its findings; the Critic noticed it as part of its meta-analysis of the expert's work. The Director then chose to pivot the investigation to focus on this issue instead." (Source: sources/2025-12-01-slack-streamlining-security-investigations-with-agents)

Canonical evidence that the weakly-adversarial-critic structure catches hallucinations and analytical gaps the expert alone would miss.

Context management architecture (series part 2)¶

The 2026-04-13 second post canonicalises how context is maintained across a long-running multi-agent investigation that can "span hundreds of inference requests and generate megabytes of output" (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications). The core architectural claim: three complementary context channels replace raw message history between agent invocations — see patterns/three-channel-context-architecture.

Channel 1: Director's Journal (structured working memory)¶

See concepts/structured-journaling-tool.

The Director has a journaling tool with six typed entry shapes. "The Director's system prompt encourages it to update the Journal often and use it for short notes. The Journal captures decisions, observations, hypotheses, and open questions in a structured format. It serves as the Director's working memory."

Six entry types (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

Type	Purpose	Example
decision	Strategic choices	"Focus investigation on authentication anomalies rather than network activity"
observation	Patterns noticed	"Multiple failed logins preceded the successful authentication"
finding	Confirmed facts	"User authenticated from IP 203.0.113.45, not in historical baseline"
question	Open items	"Was the VPN connection established before or after the suspicious activity?"
action	Steps taken/planned	"Requested Cloud Expert to examine EC2 instance activity"
hypothesis	Working theories	"This pattern suggests credential stuffing rather than account compromise"

Plus optional priority, follow-up actions, citation references. Auto-annotated with phase + round + timestamp. "The tool itself does nothing more than accumulate entries."

Every agent (Experts, Critic) receives the current Journal content in their prompt rendered as chronology. The Journal is thus shared state across all agents but authored only by the Director — reads are broad, writes are narrow. This is the canonical shared narrative that "keeps other agents on track".

Channel 2: Critic's Review (credibility scoring)¶

See concepts/credibility-scoring-rubric and patterns/critic-tool-call-introspection-suite.

The Critic reviews Expert findings in a single pass and produces an annotated findings report with credibility scores. Five-level rubric (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

Score	Label	Criteria	% of 170,000 findings
0.9-1.0	Trustworthy	Supported by multiple sources with no contradictory indicators	37.7%
0.7-0.89	Highly-plausible	Corroborated by a single source	25.4%
0.5-0.69	Plausible	Mixed evidence support	11.1%
0.3-0.49	Speculative	Poor evidence support	10.4%
0.0-0.29	Misguided	No evidence provided or misinterpreted	15.4%

Disclosed operational number: over 170,000 findings reviewed to date; sub-plausibility rate 25.8% (10.4% + 15.4%). Roughly one finding in four would reach the Director as authoritative without the Critic's filter.

The Critic has a dedicated tool-call introspection suite:

Tool	Purpose
`get_tool_call`	Inspect the arguments and metadata of any tool call
`get_tool_result`	Examine the actual output returned by a tool use
`get_toolset_info`	List what tools were available to a specific Expert
`list_toolsets`	List all available toolsets organised by Expert

These four tools let the Critic audit methodology, not just claims: did the Expert use the tool correctly, see the actual data, was the Expert properly equipped for the question (Director error), did the Expert pick the right tool. See patterns/critic-tool-call-introspection-suite.

Three mitigations against the Critic itself hallucinating (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

Stronger model tier. "Because the Critic only reviews submitted findings rather than the entire Expert run, the number of tokens required is kept within reasonable limits. While stronger models are still subject to hallucination, research suggests they err less frequently."
Narrow instructions. "The agent is instructed to only make a judgement on the submitted findings."
Downstream Timeline task as coherence check.

Channel 3: Critic's Timeline (narrative-coherence filter)¶

See patterns/timeline-assembly-from-scored-findings and concepts/narrative-coherence-as-hallucination-filter.

A separate Critic task that runs after the Review. Its inputs are three artifacts: "The most recent Review, the previous Critic's Timeline, the Director's Journal." Output is a consolidated timeline + a narrative-coherence score + top-3 gaps.

Four consolidation rules (verbatim, Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

Include only events supported by credible citations — speculation doesn't belong on the Timeline.
Remove duplicate entries describing the same event — an event shouldn't appear twice because two Experts mentioned it.
When timestamps conflict, prefer sources with stronger evidence — a log entry timestamp beats an inferred time.
Maintain chronological ordering based on best available evidence — events must flow logically in time.

Second 5-level rubric for narrative coherence (Trustworthy → Invalid). Same numeric bands as credibility, rebased from per-claim to whole-story.

Gap identification capped at top 3, three categories (see concepts/gap-identification-top-n):

Gap type	What it flags
Evidential	Missing data that would strengthen conclusions
Temporal	Unexplained periods between events
Logical	Events that don't fit the emerging narrative

Load-bearing canonical claim: narrative coherence as hallucination filter¶

"The Timeline task raises the bar for hallucinated findings by enforcing narrative coherence. To be preserved, each finding must be consistent with the full chain of evidence; findings that contradict or lack support from the broader narrative are pruned. A hallucination can only survive this process if it is more coherent with the body of evidence than any real observation it competes with."

(Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications)

This is a two-stage hallucination filter: Review scores per-finding credibility; Timeline enforces narrative consistency.

No message history between invocations¶

Verbatim (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

"Besides these resources, we do not pass any message history forward between agent invocations. Collectively, these channels provide a means of online context summarisation, negating the need for extensive message histories."

And the key argument that this is not just a token-budget optimisation:

"Even if context windows were infinitely large, passing message history between rounds would not necessarily be desirable: the accumulated context could impede the agents' capacity to respond appropriately to new information."

See concepts/no-message-history-carry-forward and concepts/online-context-summarisation.

Verbatim (Source: sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications):

"For each agent to optimally execute its role, it requires a tailored view of the investigation state. Each view must be carefully balanced. If agents are not anchored to the wider team, the investigation will be disconnected and incoherent. Conversely, sharing too much information stifles creativity and encourages confirmation bias."

Per-agent consumption map:

Agent	Journal	Review	Timeline
Director	Reads (+ writes)	Reads (for decisions)	Reads (primary summary)
Expert	Reads	—	—
Critic (Review task)	Reads	Writes	—
Critic (Timeline task)	Reads	Reads (latest)	Reads (prev) + Writes

Specimen investigation (false-positive example, 2026-04-13 post)¶

The second post includes edited specimen extracts from a real kernel-module-loading-alert investigation that turned out to be a false positive. The detection rule matched kmod in the script pathname rather than actual modprobe execution.

Timeline confidence: 0.83 (Highly-plausible band).
Experts: all four (Access / Cloud / Code / Threat) agreed.
6,046 session events retrieved by the Cloud Expert's query.
Event sequence reconstructed from 09:29:01Z → 09:31:26Z (ALERT) → 09:31:29Z (modprobe queries complete).
Three evidential gaps identified (session init timestamp unknown, triggering command not documented, secondary analyst searched wrong path).
Director's Journal shows the investigation pivoting from suspected real event → false-positive assessment over seven entries with rising priority on the concluding entries.

Forward roadmap (from the series)¶

Future posts in the series promise coverage of:

"Using artifacts as a communication channel between investigation participants" (third post, alluded to at the close of 2026-04-13)
"Human in the loop: human / agent collaboration in security investigations"
First post (2025-12-01) promised: "Maintaining alignment and orientation during multi-persona investigations" — delivered by the 2026-04-13 second post.

Caveats¶

Name inferred, not disclosed. The post calls the service "our service"; image slugs suggest spear. If a different product name is disclosed in follow-up posts, this page should be renamed.
Limited first-party numbers. Second post adds: 170,000 reviewed findings + credibility distribution + 6,046-event specimen. Still no disclosure of: throughput (investigations/day), latency per round, token cost per investigation, false-positive-rate, time-to-triage, human override rate, Critic agreement rate with human auditors.
Model families not disclosed. The three pyramid tiers are characterised only as "low, medium, and high-cost models".
Rubric dimensions partially disclosed. Second post shows the 5-level scale + distribution + criteria for each band (multi-source / single-source / mixed / poor / none). Still opaque: how the Critic decides between 0.85 and 0.90 numerically, whether there's multi-dimensional scoring (evidence × source reliability × interpretation × …) combined into one number, inter-rater reliability with humans.
Journal schema partial. Six entry types disclosed with example text, but full JSON schema, enum constraints for priority, follow-up action shape, citation ref format are not shown.
Journal mutation semantics. Append-only is strongly implied by "accumulate entries" but not explicitly stated.
MCP persistence unclear. MCP stdio mode is confirmed as prototype substrate; whether production workers speak MCP or direct data-source APIs is not disclosed.
Rollout scope unstated. No disclosure of how many alerts per day are routed to Spear vs. human investigation, or how supervision coverage scales with investigation throughput.
Second post in a continuing series. Third post promised on artifacts as inter-agent channel; this page should be extended when it publishes.

Acknowledgements (per the post)¶

Chris Smith, Abhi Rathod, Dave Russell, Nate Reeves (same team cited in both posts).

Sources¶

sources/2025-12-01-slack-streamlining-security-investigations-with-agents (series part 1, architecture + personas + pyramid + phases + service shape)
sources/2026-04-13-slack-managing-context-in-long-run-agentic-applications (series part 2, context management — Journal + Review + Timeline, credibility scoring with disclosed distribution, narrative-coherence filter, no message-history carry-forward)