--- title: Medium source: Instacart Engineering source_slug: instacart url: https://tech.instacart.com/blueberry-force-multiplier-for-the-on-call-engineer-98c446dfcc12?source=rss----587883b5d2ee---4 published: 2026-07-14 fetched: 2026-07-15T14:00:50+00:00 ingested: true --- # Blueberry: Force Multiplier For The On-Call Engineer Press enter or click to view image in full size _How we built a Slack-native on-call reasoning harness at Instacart that shortens time to first insight, speeds up theory testing, and turns tribal knowledge into reusable infrastructure._ Key Contributors: **Karthik Halukurike** , **Gabe de Oliveira, Hassan Jallad, Alan Wong** On-call work is a race to turn noisy signals into shared judgment. The hardest minutes of that race aren’t the ones spent fixing the bug — they’re the ones spent figuring out what the bug even is, while everyone in the thread is asking the same question from a slightly different angle. Blueberry is the system we built at Instacart for those several minutes. It lives in the Slack thread where the team is already coordinating, picks up each alert as it fires, and lands a grounded explanation back in the thread in about three minutes — thousands of times a month. ## On-call is mostly a clarity and speed problem Most on-call pain does not start with a major incident. It starts in the noisy window right after a page fires, when the thread fills with links, dashboards, guesses, and partial context. Someone asks whether a deployment caused it. Someone else asks how broad it is. A third engineer joins and asks what is going on. In that moment, the hardest problem is often not deep root-cause analysis. It is getting to a shared understanding quickly enough that the team can make a good next move. For us, that made two metrics matter most: time to first insight (TTFI) and time to test theories (TTTT). TTFI comes from auto-triaging every qualifying alert the moment it lands. TTTT comes from engineer-initiated on-demand investigations and follow-ups as the investigation deepens. ## Blueberry is a Slack-native on-call reasoning harness _Blueberry was built for that first window of confusion._ Blueberry lives in Slack because that is where on-call engineers already coordinate, ask clarifying questions, and make decisions under pressure. We wanted the system to work inside the shared operational conversation so the full team stays in sync in real-time and moves faster. The right mental model for Blueberry is a reasoning harness: one shared runtime that can load context, expose the right evidence surfaces, apply team-specific overlays, and produce a grounded response in the same thread where the work is happening. Press enter or click to view image in full size _The Slack thread is the runtime: thread state, team profile, skills, and memory compose into the harness, and the answer returns to the same shared thread_ ## Architecture: a durable, tool-aware reasoning harness The harness is a small number of components arranged so that any single diagnostic pass is durable, tool-aware, team-aware, and inspectable. The shape matters more than the parts, because most of what makes Blueberry sharp comes from how these pieces interact during a diagnostic pass rather than from any one of them in isolation. Press enter or click to view image in full size _Durable, tool-aware harness: production path plus side-mounted MCP catalog, persisted state, and reviewed-improvement loop_ **Ingress is selective**. Blueberry watches Slack thread events and only promotes the ones that deserve a diagnostic pass: auto-triggered alerts, @mentions, DMs, follow-ups in an active thread. The balance is between responding to too much or too little. **The job management layer is durable, not best-effort.** Diagnostic passes are enqueued onto a Postgres-backed durable queue so that they survive worker bounces, restarts, and takeovers. Multiple worker threads can drain the same queue in parallel. If a worker fails mid-pass, another picks the work up cleanly from the persisted state. For an on-call system that has to keep responding at 3am during a rolling deployment, this is a load-bearing decision. **The processor is the composition layer.** Before the agent loop starts, the processor resolves which team owns the channel, loads the team profile, layers in team-specific skills and MCP endpoints on top of the shared baseline, and prepares the tool surface that the agent will be given. Specialization happens here, not inside the agent. From the agent’s perspective, the right tools are simply present. **There are three Model Context Protocol surfaces, not one.** An in-process MCP runs alongside the agent for fast local helpers that share session state. A standalone shared MCP server hosts the heavier shared investigators, such as deploy and flag analysis, service-dependency lookup, error analysis, and anomaly sweeps, so they can be scaled and updated independently from the agent runtime. Team-hosted MCPs are mounted remotely so that data sources unique to a team show up as first-class tools without being baked into the core. The three surfaces have different latency, sharing, and ownership properties, and the processor presents them to the agent as one tool catalog. **A single diagnostic pass can fan out into parallel sub-agents.** The agent loop can spawn focused sub-agents. The alert-investigate skill uses this to launch internal knowledge searches over postmortems, runbooks, and prior incidents in parallel with live signal collection, change collection, and operational context collection. Wall-clock time is set by the slowest collector, not the sum. **Persistence captures reasoning, not just outputs.** Every diagnostic pass writes both a session-context snapshot, so the next pass or a different worker can take over without losing thread state, and a reasoning trace, so developers can later inspect how the agent decided what to do. These artifacts are what make the rest of the learning loop possible. ## Reasoning ladder: where the agent gets freedom, where we hold the line For an on-call agent, the meaningful design question is not just which model to use. It is where the system should let the agent reason dynamically, and where the harness should hold a hard line for reliability. An earlier implementation used a more workflow-shaped agent hierarchy, which was predictable but made it hard for the system to choose what to do next when an alert did not match the expected path. Moving to a more dynamic agent loop let Blueberry decide when to call tools, fan out to sub-agents, deepen into evidence, or stop and respond, while the harness kept reliability constraints around persistence, output structure, and evidence discipline. _The interesting part is the balance of freedom and structure. The model is best at choosing what to look at, weighing evidence, and deciding when an investigation is done. The harness is best at the parts that have to stay stable across thousands of diagnostic passes like capturing reasoning, keeping output predictable, and enforcing the discipline that makes the synthesis trustworthy._ ## From alert to first insight Alerts do not arrive in one clean format. They show up as monitor links, exception tracker issues, paging alerts, or raw text copied into Slack. Blueberry starts by normalizing those inputs into a shared context, optionally enriching them with team information, and then gathering evidence in parallel instead of one source at a time. _That parallelism is important._ Historical context matters, so Blueberry looks for postmortems, runbooks, and prior incidents. Live operational evidence matters too, so it checks signals, changes, and surrounding context at the same time. The goal is to move from “something fired” to a first grounded explanation as quickly as possible. Across roughly 17k auto-triggered triages in April 2026, **TTFI averaged about 3 minutes**. Press enter or click to view image in full size _Evidence fan-out: AlertContext to historical, live, team, and conditional collectors, joined into one grounded synthesis_ ## Example: SEV2 before declaration Blueberry was automatically triaging a wave of E2E test failures in a staging environment. Before the team had opened an incident channel, it identified the exact root cause: a code change had been deployed ahead of its corresponding database migration. The ORM threw a runtime error when it tried to access an undeclared enum attribute in the orders domain. Blueberry surfaced the full call stack, the deployment sequence, the dependency graph, and a clear explanation of what happened. Minutes later, the team declared the incident. The timeline below shows the important point: the first grounded explanation landed before the incident room fully formed. Press enter or click to view image in full size _Slack thread screenshot showing Blueberry’s auto-triage of the staging E2E failures landing in the channel before the incident channel was opened_ _This is the scenario Blueberry was built for: the window between the first signal and the moment the team assembles._ Getting a grounded answer into that window compresses the entire response timeline, from the first alert through remediation. ## From first insight to tested theories Getting to a first explanation is only half the job. The next question is how fast the thread can move from speculation to evidence. Blueberry helps test whether the likely culprit is a deploy, a feature flag, a dependency regression, or a recurring failure pattern that already exists in historical context. The design here is deliberate. Historical priors should inform the investigation, but live evidence should decide it. Blueberry compares the two, then narrows the next move: inspect a suspicious PR, review a flag change, look at a dependency edge, or go deeper into a trace or stack pattern if the first pass points in that direction. Across roughly 7k engineer-initiated theory checks and follow-ups in April, **TTTT averaged about 3 minutes** — about the same shape as TTFI on the original alert. _This is where the loop matters most. The agent can inspect the proposed culprit, test adjacent evidence, revise when the engineer adds context, and keep narrowing until the remaining explanation is backed by logs or code._ ## Example: 8-turn PR exoneration ending at a smoking-gun WARN log An engineer walked into a thread worried that his PR had broken staging notifications. Over eight turns, Blueberry worked through the evidence with him. It first confirmed the PR did what reviewers had asked: it converted an IllegalArgumentException to a typed NotFoundException and removed a catch-all Status.INTERNAL handler. When the pipeline failed, Blueberry traced the failure to a byte-diff between two markdown files the PR does not touch and pointed out that master was already broken pre-merge. The engineer asked whether a separate Slack report from a parallel team was related. Blueberry reframed the investigation around what could plausibly produce the reported behavior, based on what the PR actually changed. It then pulled UpdateConfig calls from mesh logs and found that the only staging error occurred three minutes after the original report, which made it timing-incompatible with the trigger. The smoking gun came from backend logs: a continuous WARN, “The event doesn’t exist or is not configured for notifications: updateDeviceConfig,” which had been firing for roughly 34 hours before the merge. Blueberry explained the design: the staging service config had updateDeviceConfig: [], an empty subscriber list by design, so the notification framework correctly logged WARN and returned before posting to Slack. It then drew the distinction in code: updateStoreConfig had subscribers and posted, while updateDeviceConfig had an empty list and stayed silent, both in ConfigManagementService.kt. ## Get Karthik Halukurike’s stories in your inbox Join Medium for free to get updates from this writer. Remember me for faster sign in The engineer had code-level proof that the gap was a pre-existing configuration, not his change. This is the long-form, multi-turn investigative arc Blueberry is designed to support: not a one-shot answer, but a careful examination backed by code and logs.