SYSTEM Cited by 1 source
Instacart Blueberry¶
Blueberry is Instacart's Slack-native on-call reasoning harness — a system that automatically triages alerts and supports multi-turn investigative conversations to compress the window between first signal and remediation.
Architecture¶
Blueberry is structured as a durable, tool-aware reasoning harness composed of:
-
Selective ingress — watches Slack thread events, only promoting qualifying ones (auto-triggered alerts, @mentions, DMs, active thread follow-ups) to diagnostic passes.
-
Postgres-backed durable job queue — diagnostic passes are enqueued durably, surviving worker bounces, restarts, and takeovers. Multiple workers drain in parallel; if one fails mid-pass, another picks up from persisted state. This is a concepts/durable-execution primitive applied to agent workloads.
-
Processor / composition layer — resolves team ownership, loads team profile, layers team-specific skills + MCP endpoints on the shared baseline, and presents a unified tool catalog. Specialization happens at composition time, not inside the agent.
-
Three MCP surfaces:
- In-process MCP — fast local helpers sharing session state
- Standalone shared MCP server — heavier investigators (deploy analysis, service-dependency lookup, error analysis, anomaly sweeps), scaled independently
-
Team-hosted remote MCPs — team-unique data sources mounted as first-class tools
-
Agent loop with parallel sub-agents — dynamic reasoning with ability to fan out into focused sub-agents for historical search (postmortems, runbooks), live signal collection, change collection, and team-contextual sources. Wall-clock time = slowest collector.
-
Dual persistence — every pass writes a session-context snapshot (for continuity) and a reasoning trace (for inspection and learning).
Design philosophy¶
"The model is best at choosing what to look at, weighing evidence, and deciding when an investigation is done. The harness is best at the parts that have to stay stable across thousands of diagnostic passes." — persistence, output structure, evidence discipline stay in the harness; reasoning freedom stays with the agent.
Operational numbers¶
- ~17,000 auto-triggered triages/month (April 2026)
- ~7,000 engineer-initiated follow-ups/month
- ~3 min average TTFI (Time to First Insight)
- ~3 min average TTTT (Time to Test Theories)