Offline Sync & Conflicts

The tap writes the local row and an outbox entry in one database transaction, and the screen updates from the database at once. Later a background job sends the op with its idempotency key and applies the server's answer in a second transaction. One local transaction: db.withTransaction { upsert row as PENDING; insert outbox op }; if either write fails neither exists, so the screen never shows an edit that will never be sent The UI reads only the database: the screen observes a Room Flow, so the edit appears instantly and survives process death; the network never writes to UI state directly Kick one unique job: enqueueUniqueWork with a network constraint; WorkManager runs it when connectivity is back, even after the app was killed Send: the worker takes the head of each chain and sends it with the Idempotency-Key minted at enqueue and the version the edit was based on Apply the answer: on success write the server's version of the row and delete the op in one transaction; on failure classify it: retry with backoff, rebase on a conflict, or park it as a failure the user can see Pull after push: then fetch the delta, so changes from other devices arrive on top of yours

An outbox row must hold everything needed to rebuild the request from data alone: an idempotency key, the target entity, the operation with a versioned payload, and the retry bookkeeping. **opId: the idempotency key, minted once at enqueue and reused by every attempt entityType + entityId: a client-generated id, so the row is addressable before the server knows it kind + payload + payloadVersion: what to do, with what data, in the format the writing build used attempts, status, baseVersion: retry state and the conflict base, all surviving process death Common mistake**: storing a built request, an auth header or a lambda; store the intent ("increment by one"), not a value from a snapshot that is gone

The row stores its own sync state as a database column, and the list is drawn only from the database. The badge then depends on data, not on what the network is doing right now. Flicker comes from two writers or a changing identity: a network callback that also pushes into UI state renders the row twice, and a key or sort field that changes on confirmation makes the item remount or jump. **syncState**: SYNCED, PENDING, SENDING, FAILED, written to the database, not to a field that process death erases **localId**: the row's identity for its whole life and the key of the list item; never swap it for the server id **createdAt: the sort key, set once, so confirmation does not reorder anything lastErrorCode: a failed row explains itself and offers Retry; only an explicit discard deletes it, with its queued ops, in one transaction Rule of thumb**: the network never touches the screen; it writes rows, and the UI is a pure function of the local database

The client mints an idempotency key once, when the payment is put in the outbox, and sends it on every attempt. The server stores the key with the first result and, on a repeat, returns that stored response instead of charging again. Retries are safe not because the client is careful, but because the server promised to deduplicate a key it durably remembers. Mint at enqueue, not in the retry loop: a key per attempt makes every retry a new payment; keep it in the outbox row and send it as the Idempotency-Key header Server: same transaction: key, status and response body are written together with the charge; recording it afterwards leaves a window where the money moved and the key did not Server: three cases: same key and payload, replay the stored response; same key, different payload, 422; same key still in flight, 409 Retention: the window must outlive the client's retry horizon, so 7–30 days is typical; an older op is reconciled against server state, not sent blind

Keep the client id as the primary key for the row's whole life and store the server id next to it in a nullable column. Rewriting the primary key is exactly what breaks foreign keys, queued payloads and list identity. The client has to mint an id because a row must be editable and referenceable before any server has heard of it: a reply created offline needs something to point at. Best case: no reconciliation: let the server accept the client id as the row's id and upsert by it; a retry is then idempotent and nothing is ever renamed **Otherwise: remoteId beside localId**: translate only at the API boundary; every local reference stays on localId Pick a sortable id: UUIDv4 is unique but unordered; ULID or UUIDv7 put a timestamp in the high bits, so rows sort by creation time and index inserts stay local Common mistake: the tidy-looking delete-and-reinsert on acknowledgement: child rows, queued mutations and list keys still hold the old id, and you have just orphaned all of them

Order the queue per dependency group, not globally: every op carries a groupKey (the root entity, here the post) and a monotonic seq inside it. The post and its comment drain in order, while unrelated groups drain in parallel. Per-aggregate FIFO: ops in one group are strictly ordered with only one in flight at a time; it is one indexed query on groupKey, status and seq **Explicit dependsOn**: when the graph is not a tree, an op names the opId it waits on and becomes eligible only after that one succeeds Resolve ids at send time: the comment's postId is a client id; look up the server mapping when sending, not at enqueue time when it does not exist yet Common mistake: one global FIFO, where a single stuck op halts every unrelated mutation in the app; the opposite mistake is an ungrouped parallel drain, where the comment gets a 404 because its post is not there yet

It deletes each op before the server has it and swallows every failure, so a crash, a timeout or a 500 loses the edit; and once the delete moves after success, the fresh UUID.randomUUID() key makes every retry apply the edit twice. **outbox.delete first**: if the process dies or push throws, the op is gone and nothing will ever resend it; delete only after success, in the same withTransaction as the upsert of the server's row **UUID.randomUUID() per send**: harmless today only because nothing is ever retried; once you delete after success, a request that landed but timed out goes again under a new key and the server applies it twice; send op.opId, minted once at enqueue **catch (e: Exception) + Result.success()**: a timeout, a 500 and a 403 look the same and none is retried; it also catches the CancellationException WorkManager uses to stop the worker. Retry transient errors with backoff, park permanent ones as visible failures **A plain loop over all(): after a failure it carries on with the next op, which may edit the row whose create just failed; stop that chain and drain the others The log line**: op.payloadJson is the user's text, now in logcat and in any crash reporter that collects logs; log the op id, type and status instead

Two triggers can start two drain() coroutines at once; each takes its own snapshot of outbox.pending(), so every op is pushed twice and the two loops interleave. Make the drain single-flight with a Mutex and read the outbox under the lock. Why it happens: the app returns to the foreground and the network comes back within the same second; each scope.launch reads the same rows before either loop has deleted anything What it costs: the idempotency key saves you only if the server deduplicates; a duplicate that lands while the first is still in flight gets a 409, and an endpoint without key support applies the edit twice How the fix works: withLock makes the second drain wait; it then reads the outbox again and finds only what is still queued, usually nothing In production: let triggers only enqueue one unique WorkManager job (enqueueUniqueWork("sync", ExistingWorkPolicy.KEEP, …)) so even two processes or a worker plus a manual drain cannot overlap Catch it: a test that fires both triggers against a fake API that suspends inside push, then asserts each opId was sent exactly once

A timestamp cursor is only correct if the server filters and orders on the same key, breaks ties deterministically and stamps rows at commit time. Otherwise page boundaries and concurrent transactions leave holes that never fill. Same key for filter and order: WHERE (updated_at, id) > (:ts, :id) ORDER BY updated_at, id; ordering by id while filtering on updated_at drops rows as soon as a page fills Deterministic tie-break: many rows share one timestamp (a whole second at second granularity); without id in the cursor a strict > skips the rest of it forever and >= loops on it Commit-time stamps: a second app server running 300 ms behind, or a transaction that read the clock before your cursor and committed after it, lands under the watermark and stays invisible for good; use a sequence assigned at commit Client side: write the rows and the cursor in one transaction, so a half-applied page cannot happen; one cursor per entity type Too-old cursor: the server answers 410 Gone and the client does a full resync instead of retrying Best of all: an opaque cursor the server issues and the client echoes back makes this the server's invariant

The protocol treated "missing from the delta" as "deleted". A delta only contains what changed, so absence means "unchanged": deletes have to travel as explicit tombstones. The server keeps the row with a deleted_at, sends it in the delta with a deleted flag, and the client applies a local delete. A client whose last sync is older than the tombstone retention window must resync fully instead of pulling a delta. Retention is a promise: the server purges tombstones after a fixed window, typically 30–90 days, and advertises it; a client that has been away longer cannot be told what it missed and must resync The two-month check: the client stores its last successful sync time; older than the window means drop the synced rows and resync from scratch, but keep the outbox, because those edits are not on the server yet. A server that knows the cursor is too old answers 410 Gone and forces the same path Tombstone contents: id, type and delete time, never the payload; that is also the honest answer to a deletion request Conflicts: a pending local edit to a row that comes back tombstoned is surfaced to the user, not silently dropped

The op carries the version it was edited from (If-Match or a baseVersion field), and the server compares it with the current one inside the write. A mismatch is rejected with 409 or 412 plus the current state, and the client rebases its intent on that state or asks the user. Where the base comes from: the server version stored on the local row at the last pull, copied into the op at enqueue; reading it at send time compares against a version the user never edited The server check is atomic: UPDATE notes SET ..., version = version + 1 WHERE id = ? AND version = ?; zero rows updated means conflict. A separate read-then-write check races with another request HTTP shape: an If-Match ETag is answered with 412 Precondition Failed, a version in the body usually with 409 Conflict; either way return the current entity so the client needs no extra round trip Client on rejection: store the server row as the new base, re-apply the pending intent on top, and auto-merge when the fields do not overlap; otherwise keep both and show a conflict. The rebased op gets the new base and a new idempotency key, because its payload changed Common mistake: retrying a 409 with backoff: waiting never changes the version, so the op fails forever and blocks its chain

Choose per field, by what the value is: last-write-wins for the title, an OR-Set CRDT for the tag list, and for the counter no merge at all: send "+1 from this user" and let the server add it up. The lesson: sync the intent, not the resulting value. Most conflicts disappear once the mutation is "add tag work" instead of "set tags to a, b and work". Last-write-wins: one version per record, the loser's row discarded. Cheap, but loses an edit when two devices touch different fields. Right for a toggle, and compare a server-assigned version, never a device clock Field-level merge: a version per field, merged column by column. Costs metadata and a merge function, still loses one of two edits to the same field, and can build a record nobody wrote: fine for a profile, wrong for money CRDT: the merge is built into the type: PN-Counter for concurrent increments, OR-Set for add and remove, sequence types for collaborative text. Costs metadata that grows with history and a real dependency (Automerge, Yjs)

Order by something the server controls or by causality you can prove (a server sequence number, a Lamport clock or a version vector), and keep wall-clock time only as a display field. The device clock answers "when did the user see this"; a sequence or a vector answers "what happened first". A fast phone breaks the merge because a field meant for display was given the job of ordering. Server sequence: a number assigned at commit, monotonic per account: a total order that is trivially correct, but two offline devices cannot order themselves against each other until they sync Lamport clock: each replica keeps a counter and on receipt sets it to max(local, received) + 1; a tie-break on device id gives a total order that never contradicts causality, but it cannot see that two edits were concurrent Version vector: one counter per replica; comparing two says A before B, B before A, or concurrent; the only one of the three that detects a conflict instead of silently overwriting Hybrid logical clock: wall time plus a Lamport counter: sorts like a timestamp, yet never goes backwards when a phone's clock does Common mistake: System.currentTimeMillis() jumps on NTP correction; SystemClock.elapsedRealtime() is monotonic but resets at boot and means nothing on another device

Never blindly upsert a row that has pending ops: store the server row as the new base and replay the pending ops on top of it, so the screen shows server state plus the user's unsent intent. Why it breaks: the pulled row is newer than your last sync but older than the local edit; a plain upsert reverts the text the user just typed, and it only comes back when the op lands and the next pull runs Base plus overlay: keep the server snapshot apart from what the UI shows; the visible row is the base with pending ops replayed in seq order. Firestore's offline cache works the same way: pending writes are overlaid on the server snapshot Cheaper variant: skip pulled rows that have pending ops and fetch them again once the outbox drains; simple, but the user sees no remote change to that row until then Push before pull: draining the outbox first narrows the window but does not close it: the user can edit in the middle of a pull Your own echo: the pull also returns rows your acknowledged ops wrote; ignore anything whose server version is not newer than the base you hold

The save merges a newer title into an op the worker is already sending; when that push succeeds, delete(op.opId) removes the row that now holds the newer title, so it is never sent and the note shows synced while the server keeps the old one. The race: the worker reads op with "Hel", the user saves "Hello", updatePayload rewrites that same row, the ack for "Hel" arrives and the delete takes "Hello" with it Why compare-and-delete is not enough: keeping the row when its payload changed and resending it under the same opId makes the server replay its stored response for "Hel" or reject it with 422; one idempotency key must mean one payload How the fix works: the worker claims the op (QUEUED to SENDING) in the transaction that reads it, and saves coalesce only into QUEUED ops; an edit made during the flight becomes a new op that goes next; a failed or interrupted send puts SENDING back to QUEUED, and resending is safe because the opId is unchanged and the payload cannot change while the op is SENDING **markSynced: only when no other op for the note is left, otherwise the badge says synced while an edit is still queued Catch it**: a test with a fake API that suspends inside putTitle, saves again during the suspension, and asserts the server ends with the last title

Only 87's own chain waits; independent chains keep draining. A timeout keeps 87 at the head to retry with backoff; a 403 is permanent, so 87 leaves the queue in one transaction, the ops that depend on it are blocked, and the user sees the edit marked as not uploaded. Transient: I/O, 408, 429, 5xx 87 stays at the head with attempts + 1 and a nextAttemptAt; nothing behind it in that chain may go, because an edit sent after its failed create gets a 404 A lost response means 87 may already be applied: resend with the same idempotency key and let the server return the original result 401 is not a reason to drop anything: refresh the token once, otherwise pause the whole queue until re-auth; 409 means rebase, not retry Permanent: 400, 403, 404, 410, 422 One transaction: delete the op, mark every later op on the same entity blockedBy it, mark the local row "not uploaded", and record a failure with the original payload Blocked dependents are released when the user retries or discards the failed edit; then blockedBy is cleared Never retry it forever (battery, a wedged chain), and never delete the payload with the op: the user watched the edit appear and considers it saved A background drain has no screen, so a toast lands nowhere; the row shows its state and one entry appears in a problems list, text still readable Why 88–200 keep going Partition the queue by aggregate root: order is a promise inside one entity's chain, not across the app, so one wedged note does not freeze the account

Queued mutations are persisted data written by a build that no longer exists, so the outbox needs its own version field and its own upgrade path. A database migration moves the table, not the meaning of the blobs inside it. Treat the outbox as a wire format between two versions of your own app and version it like an API. Self-describing rows: columns opType, payloadVersion and an opaque payload string; the table schema stays still while payload shapes evolve **Stable opType**: a string constant like "note.update", never a class name: R8 renames classes and a package move breaks the mapping silently Never persist an object graph: store a class as binary and one refactor makes every queued row unreadable Upgrade on read: decode by (opType, payloadVersion) and run the JSON through a chain of small upgrades until it matches today's shape; everything above that layer sees only the current model Dead-letter the undecodable: throwing on drain bricks sync for that user and deleting silently loses their edit; keep old decoders as long as a queue can plausibly live

Treat the photo as its own job with its own resumable upload: copy the bytes into app storage at pick time, upload them separately in chunks, and let the post's mutation carry only a reference that becomes a server blob id. Bytes move on their own schedule; ids move in the queue. Order against the post: two ops in one chain: attachment.upload(localId), then post.create(..., blobId), blocked until the first records its id; the upload op blocks only this post's chain, every other chain keeps draining Why not one multipart request: it has no resume: a timeout at 95% re-sends all 12 MB, and there is nowhere to record what already landed Resumable upload: create a session, PUT chunks with Content-Range, persist the session URL and confirmed offset after each; on resume ask the server for its offset rather than trusting your last local write Survive backgrounding: a WorkManager worker that streams from a file and calls setForeground (a dataSync foreground service, capped at about six hours a day since Android 15); if the user taps Post while online, a user-initiated data transfer JobScheduler job on Android 14+ is the alternative; nothing survives an in-memory byte array Disk: copy out of the picker's URI immediately, downscale before queueing, cap total pending attachment bytes, and delete the local copy once the blob id is stored

Every client retries and syncs on the same deterministic timer, so the whole fleet is phase-locked. The fix is randomization, not a different fixed delay: randomize anything the entire fleet would otherwise do at the same instant. Jitter in backoff: full jitter, delay = random(0, min(cap, base * 2^attempt)); plain exponential backoff keeps the herd together, because after an outage every device retries at exactly 1s, 2s, 4s **Honour Retry-After: on 429 and 503 it beats any client formula; it is the only channel the backend has for telling a million devices to spread out Cap it: a ceiling on the delay and on attempts, after which the op is parked for review instead of retried to infinity Coalesce triggers: foreground, connectivity regained, a local enqueue and the periodic floor all enqueue the same unique job; draining on every connectivity event is the 09:00 spike in miniature, because a tower coming back is one synchronized event for every device on it Spread wall-clock triggers**: if a sync is tied to a fixed time (an alarm at 09:00, a scheduled push, sync on unlock), add a stable per-install offset derived from the install id; WorkManager periodic work is anchored to its enqueue time and already spreads

Replace the clock, the network and the random source with things the test controls, run both devices and a fake server in memory, and assert invariants rather than a snapshot of the final state. A test that waits on wall-clock time measures CI load, not convergence, and that is where the red builds come from. Fake server: speaks the real endpoints, holds real state, and can be told to misbehave: drop a request, apply the write and drop the response, deliver an op twice, reorder a batch, answer 409 **One seeded Random and virtual time**: the same seed drives the fault schedule and the backoff jitter, and runTest with advanceUntilIdle() collapses hours of backoff into microseconds; print the seed so a red build reproduces Convergence: heal the network, drain, then assert deviceA.state == deviceB.state == server.state Idempotency: every op mutated server state exactly once, even though faults delivered some of them twice; this is the check people skip and the one a lost ack breaks No loss: every accepted mutation is on the server or in the failures table, and a create is never applied after its own edit; then run the same invariants over thousands of seeds

The signal is the age of the oldest unsynced write, reported as a distribution per install. A request success rate cannot see this cohort, because a wedged client eventually stops sending requests at all. Oldest pending op age: p95 and p99, sampled on every foreground; a healthy fleet sits at minutes, a stuck cohort drags the tail to days, and this is what you alert on Queue depth per install: meaningless summed across the fleet, but as a distribution it is where the 2% lives; zero completed drains per active user per day is invisible to any per-request metric Failures by class: retryable, permanent, conflict, decode failure, dead-lettered, sliced by app version, OS version and country; a 2% cohort is almost always one slice Dead-letter and clock skew: alert on any nonzero dead-letter count; device time minus the server's Date header explains a lot of "it merged wrong" reports What goes in the log: identifiers and shapes only (op id, op type, status, request id), never payload values