---
title: "We gave every blockchain its own on-call agent"
description: "How Alchemy uses 230+ agents, one per blockchain network, to triage and fix alerts across our node fleet. Median time to resolve fell from 30 minutes to 8."
---

# We gave every blockchain its own on-call agent

<ImageBlock
  src="https://media.alchemy.com/blog/on-call-agents/incident-resolution.webp"
  alt="3.5x faster incident resolution"
  width={1920}
  height={900}
  priority
/>

## How Alchemy uses 230+ agents to triage and fix alerts across our node fleet

Alchemy runs the infrastructure for 230+ blockchain networks, and operates thousands of servers that power over $1T in transactions each year. It is our business to keep every node healthy and in sync, so that the applications that depend on it stay up and running.

Over the past four months we built an agent factory that manages 230+ distinct agents, one per blockchain network, to investigate and resolve incidents. The median time to resolve an alert is now 8 minutes, down from 30 minutes for a human. The agents resolve about 73% of alerts with no human involved, and about 120 engineering hours each month now go to more productive work instead of diagnosing and fixing issues.

## The problem: 500 alerts a week across diverse infrastructure

Our node fleet produces 500+ alerts a week for our Node Ops team. Like any software engineering team, the on-call engineer takes every alert through the same five-question investigation:

1. Which network is it?
2. Are customers affected?
3. Is the problem in our fleet, or upstream in the network itself? Maybe a deploy?
4. Has this happened before?
5. What do I do next?

The answers are spread across dashboards, request logs, the alerting system, deploy history, and old postmortems. Multiply that by 500 alerts and by the idiosyncrasies of each network, and a significant amount of time goes to rebuilding context and remediating issues instead of removing causes.

<ImageBlock
  src="https://media.alchemy.com/blog/on-call-agents/fig1-alert-problem.webp"
  alt="The on-call problem: 500+ alerts a week, each taken through the same five-question investigation across scattered sources"
  width={1920}
  height={906}
/>

## The design rule: optimize for the decision

An agent with access to every tool will happily read all of them. That gives you a long report that the engineer still has to dig through. The engineer does not need more data. They need a decision: does this alert need a human, and how fast?

So we have one rule: **optimize for the decision.**

## How it works

<ImageBlock
  src="https://media.alchemy.com/blog/on-call-agents/fig2-how-it-works.webp"
  alt="How an alert moves through the agents: alert fires, find impact, find the cause, verdict, then either the agent resolves the alert or escalates to on-call"
  width={1920}
  height={982}
/>

### 1. Find impact before interrupting a human

Our alerting system already detects problems, 500+ times a week. The agents do not need to detect them again. Their job is to decide whether anyone should care.

**A firing alert is not automatically an incident.** An alert becomes an incident only when the agent finds evidence of real harm, such as customer-facing server errors, a material rise in latency, or every node in a fleet down.

If the agent finds no harm, it posts its findings to a separate "FYI" channel. Only real incidents reach the on-call channel.

### 2. Investigations with a verdict

The agent posts three to five bullets that end in one of two outcomes. If no human is needed, the agent resolves the alert itself. If a human is needed, it posts its deeper investigation to the on-call channel.

Five real examples:

<EmbeddedTable
  table={{
    columns: [
      { key: "alert", width: 180, title: "Alert", dataType: "object" },
      {
        key: "found",
        width: 360,
        title: "What the agent found",
        dataType: "object",
      },
      { key: "outcome", width: 260, title: "Outcome", dataType: "object" },
    ],
    data: [
      {
        alert: { title: "Unhealthy node", tooltip: "", icon: "" },
        found: {
          title:
            "1 of 6 nodes unhealthy, 0% server errors, a recurring condition on one isolated node",
          tooltip: "",
          icon: "",
        },
        outcome: { title: "Resolved by the agent", tooltip: "", icon: "" },
        id: 0,
      },
      {
        alert: { title: "Unhealthy node", tooltip: "", icon: "" },
        found: {
          title:
            "1 of 2 nodes unhealthy in one region, 0% server errors, P95 latency of 8 ms",
          tooltip: "",
          icon: "",
        },
        outcome: { title: "Resolved by the agent", tooltip: "", icon: "" },
        id: 1,
      },
      {
        alert: { title: "All nodes unhealthy", tooltip: "", icon: "" },
        found: {
          title:
            "The only node unhealthy, 0.14% errors, backup infrastructure is absorbing the traffic, no customer impact",
          tooltip: "",
          icon: "",
        },
        outcome: { title: "Resolved by the agent", tooltip: "", icon: "" },
        id: 2,
      },
      {
        alert: { title: "High latency", tooltip: "", icon: "" },
        found: {
          title: "22% error rate, P95 latency of 10 seconds",
          tooltip: "",
          icon: "",
        },
        outcome: {
          title:
            "Escalated; the agent proposed a severity level for a human to confirm",
          tooltip: "",
          icon: "",
        },
        id: 3,
      },
      {
        alert: { title: "Network stalled", tooltip: "", icon: "" },
        found: {
          title: "The seventh stall on the same network that day",
          tooltip: "",
          icon: "",
        },
        outcome: { title: "Escalated", tooltip: "", icon: "" },
        id: 4,
      },
    ],
  }}
/>

The engineer reads the outcome first and the evidence second, if at all.

### 3. Act inside limits that code enforces

As above, resolution often happens without a human. We gave the agents the ability to restart unhealthy nodes to restore service, and to flag configuration drift when a node's settings no longer match what we expect.

Each action has a narrow scope, and server code enforces it. An agent can:

- Restart exactly one node for a network at a time, so it cannot remove too much capacity at once.
- Pause our automatic restart system for its own network, for at most 24 hours, to avoid data corruption.
- Ask for a human. Riskier changes, such as maintenance flags or repeated restarts, need a preview and then a fresh human confirmation.

Outside those paths, the agents only read data from their scoped knowledge bases and playbooks. Scoped agent access is the name of the game when you let agents interact with live production infrastructure.

<ImageBlock
  src="https://media.alchemy.com/blog/on-call-agents/fig3-action-limits.webp"
  alt="Agent action limits enforced by server code: restart one node at a time, pause auto-restart for at most 24 hours, and require human confirmation for riskier changes"
  width={1920}
  height={1024}
/>

### 4. Make "it's upstream" the hardest conclusion

The fun part (or at least we think it is fun) of running blockchain nodes is that they are unreliable by nature. Clients can have issues, networks do not all behave the same way under different workloads, and upstream network operators (validators, sequencers, and so on) can cause our infrastructure to stall or fail. So every agent needs _network-specific memory and instructions_, and it needs them to answer one question: "Is it us, or is it them?"

An agent must meet four conditions before it says "upstream":

1. Our nodes must be healthy, from an error-rate and latency perspective.
2. The upstream errors (for example, a network "stall") must look the same in every region we run.
3. Independent providers must confirm it, and customer requests must otherwise look normal.
4. A stall in only some regions is never upstream.

The rule works in both directions. In one alert, six nodes were unhealthy, but the network was still producing new blocks. The agent called it our serving problem, not an upstream stall.

<ImageBlock
  src="https://media.alchemy.com/blog/on-call-agents/fig4-upstream-rule.webp"
  alt="The four conditions an agent must meet before it calls a problem upstream"
  width={1920}
  height={909}
/>

## Results: higher reliability, faster resolution

Because of these investments, engineering hours go to meaningful work, and our customers get faster resolution of issues on any network they build on.

- **Alerts resolve almost four times faster.** Median time to resolve fell from 30 minutes to 8.
- **Most alerts never need an engineer.** The agent resolves about 73% of alerts on its own, up from 0% in June.
- **The team got about 120 hours a month back.** Engineering time on alerts fell from about 150 hours a month to about 30, an 80% reduction. That is about three engineer-weeks each month that now go to removing causes instead of rebuilding context.
- **The agent is right the first time.** Across 200+ investigations in the past month, first-try root-cause accuracy was about 98%.

<ImageBlock
  src="https://media.alchemy.com/blog/on-call-agents/fig5-impact.webp"
  alt="Impact from June to October: median time to resolve 8 minutes, about 73% of alerts resolved with no human, about 30 engineering hours a month, about 98% first-try accuracy"
  width={1920}
  height={1181}
/>

## What we would tell a team building its own on-call agent

1. **Measure outcomes.** Judge an on-call agent by time to resolve, accuracy, and engineering hours saved.
2. **Ask for a verdict.** An agent that hands back a page of findings has moved the work, not removed it. Require it to lead with a decision: resolve it, or page a human and say how urgently.
3. **Grant named actions with fixed scopes.** A single-node restart with a 24-hour cap is safer than raw shell access.
4. **Give each agent a narrow beat and a long memory.** One agent per network, with that network's runbooks, quirks, and incident history, does not have to rediscover your system on every alert. It already knows this is the seventh stall today.

But most importantly: **do not stop at issue investigation.** An agent that can _prevent_ issues from ever reaching a human compounds the engineering effort you spend on customer value.

Like this kind of crossover between applied AI and infrastructure?

- Follow [@onewaysidewalks](https://x.com/onewaysidewalks) to stay up to date on all things AI at Alchemy.
- [Join the team](https://www.alchemy.com/careers). We're hiring!