Sync engines
The reconnect storm problem
When 800 phones come back online at once, what happens to your server. The patterns that survive.
Loading essay…
When 800 phones come back online at once, what happens to your server. The patterns that survive.
Loading essay…
A reconnect storm is what happens when many devices come back online at the same moment. Power outage in a city, a router reboot at a chain restaurant, a tower handover that resets thousands of cellular sessions: all of them produce the same shape. A flood of clients, each carrying a queue, each opening a websocket, each issuing the same first request. The server, which was idle a second ago, is now under more load than it sees in a normal week.
This essay is about what we did the first time it broke us, and the patterns that worked.
It is worth being precise about what is happening. The clients are not malicious; they are doing what the docs told them to do. On reconnect:
In normal operation these steps are spread across users and time, so the server sees a steady trickle. In a storm, a few thousand devices fire all five steps within the same five-second window. The server spends those five seconds servicing handshakes; the database spends them servicing 50,000 small queries with high lock contention; the websocket layer spends them spawning 5,000 new connections. The slowest step gates the whole thing.
The first time it happened to us, the server stayed up but the queries queued; clients started timing out, started retrying, and the retries made the storm worse. By minute three we were watching the queue grow faster than we could drain it. By minute five we hand-rolled a kill switch.
We did not get them all in one go. We got the first one in panic that afternoon and the rest over the following month.
The single biggest win. Every client, on losing connection, picks a
random delay in [0, jitter_max] before its first reconnect attempt;
on subsequent failures the delay grows exponentially with another
small jitter. This breaks the synchronisation. A thousand clients no
longer reconnect at second 0; they spread across [0, 30], and the
server sees a slope instead of a wall.
Duration nextReconnectDelay({required int attempt}) {
final base = math.min(30000, 500 * math.pow(1.7, attempt).toInt());
final jitterMs = math.Random().nextInt(base ~/ 4);
return Duration(milliseconds: base + jitterMs);
}Note: jitter has to be multiplicative-ish, not additive. Adding ±200 ms to a 30 s delay barely smooths the curve; multiplying or adding a noticeable fraction does.
We had inherited per-entity sync endpoints (/products/list,
/orders/list, /contacts/list, …). On reconnect a single client
issued 48 of these. A unified pull endpoint that takes a since
cursor across all entity types in one request collapses 48 requests
into 1. Fewer connections, fewer queries, less ceremony.
POST /api/v1/sync/pull
{
"cursors": {
"contacts": "2026-05-08T14:22:01.123Z",
"invoices": "2026-05-08T14:22:01.500Z",
"products": "2026-05-08T14:22:00.000Z"
},
"limits": { "perEntity": 200 }
}The server returns a single bundle keyed by entity type. The client walks the bundle and applies in dependency order (contacts before invoices, products before invoice items, etc).
If one tenant is misbehaving (or has a thousand devices), they should not starve the others. Instead of a single global Horizon queue, we shard by tenant and round-robin between active tenants. A noisy tenant takes longer to drain its own queue; its noise does not bleed across.
The trade-off is that you cannot use Horizon's autoscaling per "workload" any more, since "workload" is now per tenant. Worth it.
The pull endpoint is the cold path. If it starts failing more than
~5 % over a sliding minute, we trip a breaker on the server side and
return a polite "try again in 30 seconds" with a Retry-After header.
The clients honour it. The server gets its breath back. Without the
breaker, every failed pull triggers a retry, the retry adds load, the
load triggers more failures.
The websocket layer (Reverb in our case, but it could be any) has a finite per-process connection budget. Without limits, a storm fills the pool, new connections get refused at the OS level, and the error message at the client end says "Connection refused" — useless. Setting explicit per-tenant connection caps and per-process maximums means the server starts shedding cleanly, with a header that tells the client why.
The dashboard for storm health is small:
| Metric | Healthy | Warning |
|---|---|---|
| Reconnects per second | < 50/s | > 200/s |
| Pull p99 latency | < 800 ms | > 3 s |
| Queue drain rate | > 1×in-flow | < 0.7×in-flow |
| Breaker open events / hour | 0 | ≥ 1 |
| Pool utilisation | < 60 % | > 85 % |
The interesting one is "queue drain rate ÷ in-flow rate." If it is below 1, you are losing ground. If it stays below 1 for more than a minute or two, the storm is winning. The kill switch trips somewhere around 0.6.
The first time we had a storm, the team got pulled into an unplanned incident on a Tuesday morning that ended up costing about six engineer hours and a panicked apology to one customer. The five fixes above took roughly two weeks total to design, ship, and validate. The next storm, the system handled itself and we found out about it from the dashboard, not the customer. That is the trade you are buying.
If sync engines are interesting, the series page picks up where this ends.