Sync engines
Building a sync engine, part one: the queue
Series opener. The shape of an outgoing queue, why ordering matters, and the test harness that catches everything.
Part ofsync-engineLoading essay…
Series opener. The shape of an outgoing queue, why ordering matters, and the test harness that catches everything.
Part ofsync-engineLoading essay…
A sync engine is a small queue, two endpoints, and a lot of edge cases. The queue is where most engines fail in their first month, because the obvious implementation works perfectly for the first hundred users and then quietly loses data. This is part one of five on the engine that keeps a Flutter client and a Laravel backend honest. We start with the queue because every other piece of the engine assumes it works.
Every offline-capable write the user makes (creating an invoice, editing a contact, deleting a row) lands in a local table on the device first, then joins a queue waiting to be pushed to the server. The queue must:
If your queue does not do all four, you do not have a queue. You have an optimistic write and a prayer.
After three rewrites we landed on this row shape, in a Drift table on the client:
@DataClassName('SyncQueueItem')
class SyncQueue extends Table {
TextColumn get id => text()(); // local UUID
TextColumn get entityType => text()(); // 'invoice', 'contact'
TextColumn get localEntityId => text()(); // the row this writes
TextColumn get operation => text()(); // 'create' | 'update' | 'delete'
TextColumn get payload => text()(); // JSON, the actual diff
TextColumn get idempotencyKey => text().unique()(); // server contract
IntColumn get attempts => integer().withDefault(const Constant(0))();
TextColumn get status => text() // 'pending' | 'in_progress' | 'failed' | 'synced'
.withDefault(const Constant('pending'))();
DateTimeColumn get createdAt => dateTime()();
DateTimeColumn get nextAttemptAt => dateTime().nullable()();
TextColumn get lastError => text().nullable()();
@override
Set<Column> get primaryKey => {id};
}Three details that took bug reports to learn:
idempotencyKey is the server's contract, not yours#Every push request sends an Idempotency-Key header. The server stores it
keyed to the resulting row id. If the same key arrives twice, the server
returns the original response without re-executing. This is the only way
to survive a phone that pushes, the response gets lost, and the phone
retries.
We generate the key on the client at queue-insert time, not at push time. Generating it later means a retry might hash differently and the server treats it as a new request.
status has four states, not two#You will be tempted to use pending and done. You need in_progress
because a phone can crash mid-push, and failed because some errors are
permanent (validation errors, deleted parents) and should not be retried
forever. Without these two states the queue either retries permanent
errors in a loop or drops valid items on a temporary one.
nextAttemptAt is set, not computed#We learned this twice. The retry interval cannot be computed at scan time ("if attempts > 0 wait N seconds"); it has to be stored when the failure happens, with jittered backoff baked in. Otherwise every device on the same broken network retries at the same instant, every interval, until they succeed or the server falls over. We had this happen on a Tuesday morning. It is not fun.
The push loop runs every few seconds while the app is foregrounded:
Future<void> pushOnce() async {
final batch = await db.queueRepo.claimBatch(
max: 20,
includeStatuses: const ['pending'],
where: 'next_attempt_at IS NULL OR next_attempt_at <= ?',
);
if (batch.isEmpty) return;
// Mark every claimed item in_progress in one transaction so a crash
// here does not strand them in a half-claimed state.
await db.queueRepo.markInProgress(batch.map((b) => b.id).toList());
try {
final response = await api.batchPush(batch);
await db.queueRepo.applyPushResponse(batch, response);
} catch (e) {
// Network blip. Reset the batch to pending with backoff.
await db.queueRepo.resetForRetry(batch.map((b) => b.id).toList());
}
}Two lines of that loop are non-obvious:
The other one is the claimBatch query: it must filter next_attempt_at
inside the same transaction that flips status, otherwise two concurrent
push loops can claim the same item twice.
Here is the scenario that breaks naive engines:
local_id = uuid-A.contact_id = uuid-A in the invoice payload.server_id = 4729.uuid-A.The fix is to rewrite the FK at push time, after the parent has been
pushed and we know its server id. Walk the payload, find every field
ending in _id, look up the local-to-server mapping for that entity, and
substitute. We have a small whitelist of FK fields per entity to avoid
touching anything else.
function resolvePayload(payload: any, entityType: string): any {
const fkFields = FK_MAP[entityType] ?? [];
const out = { ...payload };
for (const field of fkFields) {
const localId = out[field];
if (typeof localId === 'string' && localId.startsWith('local-')) {
const serverId = serverIdMap.get(localId);
if (serverId == null) {
throw new ParentNotSyncedError(field, localId);
}
out[field] = serverId;
}
}
return out;
}If the parent has not synced yet (the lookup misses), we throw and skip the row this round. The next round, the parent will have its server id, and the child can go through.
This part of the engine is structurally simple. The next part, ordering, is where the abstractions start earning their keep: when two clients disagree about the same row, who wins, and how do you make the answer both correct and explainable to the user. That is part two.
The whole series page tracks what we have shipped and what is still being written.