INSTACART

Blueberry: Force Multiplier For The On-Call Engineer¶

Summary¶

Instacart built Blueberry, a Slack-native on-call reasoning harness that automatically triages alerts and supports multi-turn engineer-initiated investigations. It lives in Slack threads where teams already coordinate, picks up qualifying alerts, gathers evidence in parallel from multiple sources (postmortems, runbooks, live signals, deploy history), and returns a grounded explanation within ~3 minutes. In April 2026, it auto-triaged ~17,000 alerts and served ~7,000 engineer-initiated theory checks.

Key takeaways¶

Two north-star metrics: Time to First Insight (TTFI) and Time to Test Theories (TTTT) — both average ~3 minutes. The hardest on-call minutes are spent building shared understanding, not fixing the bug.
Durable, not best-effort job management: Diagnostic passes are enqueued onto a Postgres-backed durable queue that survives worker bounces, restarts, and takeovers. Multiple workers drain in parallel; failed workers are picked up cleanly from persisted state. "For an on-call system that has to keep responding at 3am during a rolling deployment, this is a load-bearing decision." (Source: sources/2026-07-14-instacart-blueberry-on-call-reasoning-harness)
Three MCP surfaces, not one: (1) In-process MCP for fast local helpers sharing session state; (2) standalone shared MCP server for heavier investigators (deploy analysis, service-dependency lookup, error analysis, anomaly sweeps) scaled independently; (3) team-hosted remote MCPs for team-unique data sources. The processor presents all three as one unified tool catalog to the agent.
Team-specific specialization at composition time: Before the agent loop starts, the processor resolves channel ownership → loads team profile → layers team-specific skills + MCP endpoints on shared baseline → presents the tool surface. "Specialization happens here, not inside the agent."
Parallel sub-agent fan-out: A single diagnostic pass can spawn focused sub-agents. The alert-investigate skill launches internal knowledge searches, live signal collection, change collection, and operational context collection in parallel. Wall-clock time = slowest collector, not sum.
Selective ingress: Blueberry watches Slack thread events and only promotes those deserving a diagnostic pass — auto-triggered alerts, @mentions, DMs, follow-ups in active threads. Balances over-responding vs. under-responding.
Persistence captures reasoning, not just outputs: Every diagnostic pass writes (a) a session-context snapshot for continuity and (b) a reasoning trace for post-hoc inspection. These artifacts enable the learning/improvement loop.
Harness holds structure, agent holds reasoning: The design separates what the model is good at (choosing what to look at, weighing evidence, knowing when to stop) from what the harness must keep stable (persistence, output structure, evidence discipline). Moved from a workflow-shaped hierarchy to a dynamic agent loop for flexibility.
Historical priors inform, live evidence decides: The system compares historical context (postmortems, runbooks, prior incidents) with live operational signals, then narrows the investigation based on where evidence actually points.
Production example — SEV2 caught before incident declaration: Blueberry identified that E2E test failures were caused by a code change deployed ahead of its DB migration (ORM hit undeclared enum attribute), surfacing the full call stack, deployment sequence, and dependency graph before the incident room even formed.

Operational numbers¶

Metric	Value
Auto-triggered triages (April 2026)	~17,000
Engineer-initiated follow-ups (April 2026)	~7,000
TTFI (average)	~3 minutes
TTTT (average)	~3 minutes

Architecture¶

Slack thread events
    ↓ (selective ingress filter)
Postgres-backed durable job queue
    ↓ (parallel worker drain)
Processor / composition layer
    ├─ resolve team → load team profile
    ├─ layer team-specific skills + MCP endpoints
    └─ present unified tool catalog
         ↓
Agent loop (dynamic reasoning)
    ├─ in-process MCP (fast local helpers)
    ├─ shared MCP server (deploy/error/anomaly)
    ├─ team-hosted remote MCP (team-unique sources)
    └─ parallel sub-agents:
         ├─ historical: postmortems, runbooks, prior incidents
         ├─ live: signals, traces, errors
         ├─ changes: deploys, flags
         └─ team-contextual: team-specific sources
              ↓
    Synthesis → grounded explanation in Slack thread
    Persistence: session-context snapshot + reasoning trace

Systems / concepts / patterns extracted¶

Systems: systems/instacart-blueberry

Concepts: concepts/durable-execution, concepts/context-engineering, concepts/sub-agent, concepts/on-call-automation, concepts/time-to-first-insight, concepts/time-to-test-theories, concepts/reasoning-trace, concepts/multi-surface-mcp, concepts/selective-ingress, concepts/incident-mitigation-lifecycle

Patterns: patterns/parallel-subagent-execution-for-latency, patterns/durable-job-queue-for-agent-reliability, patterns/multi-surface-mcp-composition, patterns/harness-holds-structure-agent-holds-reasoning, patterns/evidence-fan-out-then-synthesize

Caveats¶

Instacart is Tier 2 but this is a Tier-3-shaped source (Medium post, not tech.instacart.com domain proper) — however the architectural depth (durable queue design, MCP tri-surface, fan-out parallelism, worker failover) is substantial.
No latency percentiles disclosed beyond "about 3 minutes" averages.
The post focuses on the "first window" use case; long-running incident management beyond the initial triage is not addressed.
No discussion of cost, infrastructure sizing, or failure modes of the system itself.