How we rebuild node fleets 15x faster

Most of our reliability work is invisible by design. You send an RPC request, you get an answer in a few milliseconds, and you never think about the machine that served it. That is exactly how it should feel. But behind that one request sits a fleet of blockchain nodes that has to stay healthy around the clock across many chains, in multiple regions, without pause. Keeping that fleet healthy at scale turns out to be a data-movement problem, and a much bigger one than most people expect.

This post is about one specific investment we made to solve it: a high-bandwidth, parallelized data-transfer system we built in-house, paired with a dedicated private network between our machines. Together they let us rebuild node storage in a fraction of the time it used to take. This is first and foremost a reliability investment, and reliability is what keeps latency low and consistent when hardware fails. It is one deliberate piece of the availability guarantees our customers build on with our node infrastructure, not the whole story of how we defend latency.

The honest version of the claim, up front: this does not make an individual RPC call faster on a normal day, because it is not in your request path. What it does is make recovery fast and reliable exactly when a machine fails, the moment your latency is most at risk. So yes, it supports low latency, in the edge cases where it counts.

What are we actually recovering?

Worth answering directly, because "recovery" can mean a few things. This is about restoring node state (the on-disk blockchain dataset a node needs to answer requests) onto healthy hardware. It is not about recovering an individual failed request, in-flight data, or the physical hardware itself.

We reach for it in three situations:

• A node fails or degrades. The hardware or its disk is unhealthy, so we bring up a replacement and need to load the full chain state onto it fast.
• We add capacity. Traffic for a chain grows and we need more healthy nodes serving it, each needing the complete dataset before it can take traffic.
• We bring up a new chain or region. New nodes start empty and need terabytes of state before they are useful.

In all three, the bottleneck is the same: getting a large, ever-growing dataset onto fresh hardware quickly and reliably.

Why is rebuilding a blockchain node a data-movement problem?

A blockchain full node or archive node is not a small thing. Depending on the chain, its on-disk state runs from hundreds of gigabytes into the multi-terabyte range and it grows every single day. Every time we hit one of the situations above, we have to get that state onto fresh hardware, fast.

There are two slow ways to do that. You can sync from the network, letting the node replay history from its peers, which can take days for a large chain. Or you can copy the state from a healthy node you already run. Copying is dramatically faster than syncing from scratch, so that is the path we optimize.

The catch: copying terabytes is still slow if you do it one node at a time, over a shared network path, with a tool that was never built for this scale.

What is rsync, and why did we outgrow it?

We did not start by building something new. For years we used rsync, the standard Unix tool for copying and synchronizing files between machines.

What it is. rsync copies files from one machine to another and is smart about repeat runs: it uses a rolling-checksum delta algorithm to transfer only the parts of a file that changed, rather than recopying everything. It is battle-tested, ubiquitous, and its delta-transfer approach makes incremental copies cheap. For syncing a dataset between two machines, it is hard to beat.

Where it broke down for us. rsync was designed to sync files between two hosts, not to move terabytes onto fresh hardware under time pressure. At our scale it ran into a hard limit: throughput. A single rsync stream is largely serial and does not come close to saturating the fast network and storage hardware we run, so much of the available bandwidth sits idle. At fleet scale that wasted capacity compounds into hours of rebuild time per node and real operational drag. rsync was not wrong. We simply outgrew what it was built to do. So we built a transfer tool designed from the start for one job: moving enormous datasets onto fresh nodes as fast as the hardware allows.

How we do it: a dedicated private network plus a parallel transfer tool

We attacked the problem on two fronts at once: the road and the vehicle.

The road: a dedicated private network

We moved bulk node-data transfer onto a dedicated private network between our machines, separate from the path that serves production traffic.

Two things change the moment the copy runs on that private network. First, bulk transfer no longer competes with production request traffic, so a large rebuild never puts customer-facing latency at risk. Second, it delivers high, predictable bandwidth between nodes: the raw capacity the transfer tool is built to use, up to tens of gigabits per second.

The vehicle: a purpose-built parallel transfer tool

Our in-house tooling does one job: push a full directory of node state from a healthy source machine to a destination machine as fast as possible. The design is built around saturating a high-bandwidth link that a single serial copy would leave mostly idle. Three ideas do the work:

• Many parallel connections. Instead of one stream, the transfer runs across many concurrent TCP connections at once, so it can actually fill the private network's bandwidth.
• Splitting very large files. A single multi-terabyte state file would bottleneck one connection, so files above a configurable size threshold are split into chunks by byte offset, and each chunk is sent over its own connection in parallel. The destination writes each chunk straight to the right offset in the file.
• Batching small files and going biggest-first. Many small files are grouped together to amortize per-transfer overhead, and the largest files start first so the longest transfers are underway immediately.

The orchestration: Ansible and AWX

A fast transfer tool is only useful if you can drive it reliably across many machines without a human typing commands. We orchestrate rebuilds with Ansible, which runs defined tasks across many servers, managed through AWX, its job-and-scheduling control plane.

The payoff is that a rebuild becomes a repeatable, auditable job: select the target, kick off the workflow, and let the system carry the state across the private network in parallel - with consistent results every time.

How much faster is our solution than rsync?

Our tooling is 10-15x faster. Single-threaded rsync moves node state at 100-200 MB/s. Our parallelized transfer sustains 2-3 GB/s.

That speed shows up where it counts, in a real rebuild. In one production run we rebuilt a Base Mainnet fleet of 13 nodes (one source, twelve destinations) with the actual data transfer completing in about 35 minutes, and the full workflow including private-network setup and teardown finishing in roughly 1 hour 27 minutes, with zero failures and zero unreachable nodes. The old node-by-node approach turned the same kind of rebuild into a multi-hour slog.

What this unlocks for you with Alchemy

This is a foundation-layer capability, so its benefit reaches any product running on our node fleet. The mechanism is always the same (a hardware failure heals before it can concentrate load and push tail latency up), but here is what that means for you, product by product.

• Node & RPC APIs (Alchemy's RPC API): your p95/p99 response times stay steady through a node failure, because a degraded node is replaced before the survivors get overloaded. Fewer slow-request spikes for your users.
• Data & indexing APIs (Transfers, Balances, Token, Webhooks): fewer gaps and less lag in the data you consume when a backing node is rebuilt, because recovery is a short window instead of hours. More trustworthy data, sooner.
• Rollups & chain services: faster capacity restoration for your dedicated or app-specific chain, so a single hardware fault does not turn into a prolonged degraded window for your app.

These are real consistency gains: we keep your experience fast by making the difficult moments short and rare.

Why this matters for the people who never see it

Here is the connection back to your application. When a node degrades or fails, its traffic has to shift to the remaining healthy nodes. If rebuilding a replacement is slow, that pressure sits on the survivors longer, and concentrated load is precisely what drives tail latency up: the p95 and p99 response times that surface as your slowest requests. (The classic treatment of this effect is Dean and Barroso's The Tail at Scale.) Fast recovery shrinks that window. The fleet heals before the degradation is ever felt as a slow response on your end.

This is what we mean when we say we plan for failure. We do not assume machines stay up. We assume they will fail, and we invest so that failure stays small and recovery stays fast. This transfer system and the private network behind it are one of those investments: infrastructure whose entire job is to make sure a hardware problem in our data center never becomes a latency problem in your app.

You will probably never notice any of this. That is the point.

The bigger picture: fast, and fast under stress

It is easy to post a good latency number on a healthy day. The harder claim, the one that actually matters in production, is staying fast when things break. Low latency is not just a steady-state property. It is a property of how a system behaves during failure, and how quickly it recovers. It is also why we run heterogeneous, redundant node infrastructure in the first place.

This is one part of how we defend that: a deliberate investment in the recovery path, so the response times you rely on hold up not just when everything is green, but when a machine goes down at 3 a.m. and has to be rebuilt before anyone notices.

That is the standard we hold ourselves to. It is why performance work at Alchemy goes all the way down to how fast we can move a terabyte between two machines.

Frequently asked questions

Does this make my RPC requests faster?

On a normal day, no. It is not in your request path. But it protects your latency in the moment that matters most: when hardware fails. By rebuilding a healthy node fast, it keeps concentrated load from driving up your p95/p99, so your slowest requests stay fast under stress.

What exactly does it recover?

Node state - the on-disk blockchain dataset a node needs to serve requests. We use it to replace a failed node, add capacity, or bring up a new node or region. It does not recover individual requests or physical hardware.

Why not just use rsync?

rsync is excellent for syncing files between two machines, and we used it for years. But a single rsync stream is largely serial, so at fleet scale it does not saturate our network and storage hardware, adding up to hours of rebuild time per node. Our tool runs the transfer across many parallel connections and splits very large files across them to use the full bandwidth of a dedicated private network.

How does faster node recovery improve latency?

When a node fails, its load shifts to the remaining healthy nodes. The longer a rebuild takes, the longer that concentrated load drives up tail latency: the p95 and p99 response times users feel as their slowest requests. Faster recovery shrinks that window so failures do not surface as slow responses.

How much faster is it than rsync?

About 10-15x on raw transfer throughput: roughly 100-200 MB/s for a single-threaded rsync stream versus 2-3 GB/s for our parallelized transfer over the private network. In practice that turned a multi-hour fleet rebuild into one where the data transfer finished in about 35 minutes, across a 13-node Base Mainnet fleet with zero failures.

Want to build on infrastructure engineered this way? Explore the docs at alchemy.com/docs.