Walk through an offline-first write: what happens between the user tapping Save and the server confirming it?

The tap writes the local row and an outbox entry in one database transaction, and the screen updates from the database at once.

A mutation has sat in the outbox for three weeks across two app updates. What must that row have stored for it to still be replayable?

An outbox row must hold everything needed to rebuild the request from data alone: an idempotency key, the target entity, the operation with a versioned payload, and the retry bookkeeping.

A payment POST times out before the client sees a response, so the client retries. What keeps that from charging twice, and who mints the key?

The client mints an idempotency key once, when the payment is put in the outbox, and sends it on every attempt.

You create a row offline with a client UUID and the server returns its own id on first sync. How do you reconcile them without breaking every reference?

Keep the client id as the primary key for the row's whole life and store the server id next to it in a nullable column.

A queued comment refers to a post that is itself still sitting unsynced in the outbox. How does the queue guarantee the post goes first?

Order the queue per dependency group, not globally: every op carries a `groupKey` (the root entity, here the post) and a monotonic `seq` inside it.

Your pull sends ?updated_since= and users report rows that simply never arrive. What must the server guarantee for that cursor to be correct?

A timestamp cursor is only correct if the server filters and orders on the same key, breaks ties deterministically and stamps rows at commit time.

A phone comes back after two months offline and still shows notes that everyone else deleted. What did the sync protocol get wrong?

The protocol treated "missing from the delta" as "deleted".

Your queued edit was based on version 5, but the server is already at version 7. How does the server notice, and what does the client do with the rejection?

The op carries the version it was edited from (`If-Match` or a `baseVersion` field), and the server compares it with the current one inside the write.

For a note title, a like counter and a shared tag list, which of last-write-wins, field-level merge and a CRDT do you pick, and what does each cost?

Choose per field, by what the value is: last-write-wins for the title, an OR-Set CRDT for the tag list, and for the counter no merge at all: send "+1 from this user" and let the server add it up.

Your merge keeps picking the wrong winner because one phone's clock is forty minutes fast. What do you order events by instead?

Order by something the server controls or by causality you can prove (a server sequence number, a Lamport clock or a version vector), and keep wall-clock time only as a display field.

A delta pull arrives while the user's edits to the same rows are still in the outbox. How do you apply it without wiping those edits?

Never blindly upsert a row that has pending ops: store the server row as the new base and replay the pending ops on top of it, so the screen shows server state plus the user's unsent intent.

You ship an update that changes the shape of a mutation payload, and thousands of users still have old-format ops sitting in their outbox. What now?

Queued mutations are persisted data written by a build that no longer exists, so the outbox needs its own version field and its own upgrade path.

A post created offline has a 12 MB photo attached. How do you get the bytes up without blocking the mutation queue or filling the user's disk?

Treat the photo as its own job with its own resumable upload: copy the bytes into app storage at pick time, upload them separately in chunks, and let the post's mutation carry only a reference that becomes a server blob id.

Sync has been quietly broken for 2% of your users for a month. Which numbers would have told you?

The signal is the age of the oldest unsynced write, reported as a distribution per install.

Irbisa · Cheatsheet

Offline Sync & Conflicts

Q: Find the bug: when the app returns to the foreground just as the network comes back, the server log shows every queued edit pushed twice.

Two triggers can start two `drain()` coroutines at once; each takes its own snapshot of `outbox.pending()`, so every op is pushed twice and the two loops interleave.

Mutation outbox, idempotency keys, delta sync, tombstones, LWW vs CRDT, backoff, convergence

Senior Developer · 21 questions

01
Walk through an offline-first write: what happens between the user tapping Save and the server confirming it?
The tap writes the local row and an outbox entry in one database transaction, and the screen updates from the database at once.
02
A mutation has sat in the outbox for three weeks across two app updates. What must that row have stored for it to still be replayable?
An outbox row must hold everything needed to rebuild the request from data alone: an idempotency key, the target entity, the operation with a versioned payload, and the retry bookkeeping.
```
@Entity(tableName = "outbox")data class PendingOp(  @PrimaryKey val opId: String,      // idempotency key: minted once, never regenerated  val groupKey: String,              // aggregate root: the ordering scope  val seq: Long,                     // monotonic within the group  val entityType: String,…
```

What does a locally created row carry so the list can show sending and failed without the row flickering or jumping?

The row stores its own sync state as a database column, and the list is drawn only from the database.

enum class SyncState { SYNCED, PENDING, SENDING, FAILED } @Entity(tableName = "notes")data class NoteEntity(  @PrimaryKey val localId: String,      // stable for the row's whole life  val remoteId: String? = null,         // arrives later, never becomes the key…

04
A payment POST times out before the client sees a response, so the client retries. What keeps that from charging twice, and who mints the key?
The client mints an idempotency key once, when the payment is put in the outbox, and sends it on every attempt.
```
interface OrdersApi {  @POST("orders")  suspend fun create(    @Header("Idempotency-Key") key: String,    @Body body: CreateOrder  ): Response<Order>…
```
05
You create a row offline with a client UUID and the server returns its own id on first sync. How do you reconcile them without breaking every reference?
Keep the client id as the primary key for the row's whole life and store the server id next to it in a nullable column.
```
@Entity(tableName = "posts")data class PostEntity(  @PrimaryKey val localId: String,   // ULID minted on the device, never rewritten  val remoteId: String? = null,      // filled in on the first successful sync  val title: String)…
```
06
A queued comment refers to a post that is itself still sitting unsynced in the outbox. How does the queue guarantee the post goes first?
Order the queue per dependency group, not globally: every op carries a groupKey (the root entity, here the post) and a monotonic seq inside it.
```
@Daointerface OutboxDao {  @Query("SELECT DISTINCT groupKey FROM outbox WHERE status = 'READY' AND nextAttemptAt <= :now")  suspend fun readyGroups(now: Long): List<String>   // the head of one group: strictly one op in flight, in seq order…
```

This worker drains the outbox. Which lines lose a user's edit or apply it twice, and how do you fix them?

class SyncWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, params) {  override suspend fun doWork(): Result {    for (op in outbox.all()) {      outbox.delete(op.opId)      try {        val saved = api.push(UUID.randomUUID().toString(), op.payloadJson)        notes.upsert(saved.toEntity())      } catch (e: Exception) {        Log.w("sync", "failed: ${op.payloadJson}")      }    }    return Result.success()  }}

It deletes each op before the server has it and swallows every failure, so a crash, a timeout or a 500 loses the edit; and once the delete moves after success, the fresh UUID.randomUUID() key makes every retry apply the edit twice.

Find the bug: when the app returns to the foreground just as the network comes back, the server log shows every queued edit pushed twice.

class SyncManager(  private val outbox: OutboxDao,  private val api: NotesApi,  private val scope: CoroutineScope,) {  fun onNetworkAvailable() { scope.launch { drain() } }  fun onAppForeground() { scope.launch { drain() } }   private suspend fun drain() {    for (op in outbox.pending()) {           // ordered by seq      api.push(op.opId, op.payloadJson)      outbox.delete(op.opId)    }  }}

Two triggers can start two drain() coroutines at once; each takes its own snapshot of outbox.pending(), so every op is pushed twice and the two loops interleave.

09
Your pull sends ?updated_since=<last timestamp> and users report rows that simply never arrive. What must the server guarantee for that cursor to be correct?
A timestamp cursor is only correct if the server filters and orders on the same key, breaks ties deterministically and stamps rows at commit time.
```
@Serializabledata class Cursor(val updatedAt: Long, val id: String)   // value + tie-break @Serializabledata class DeltaPage(val rows: List<NoteDto>, val next: Cursor?, val hasMore: Boolean)…
```
10
A phone comes back after two months offline and still shows notes that everyone else deleted. What did the sync protocol get wrong?
The protocol treated "missing from the delta" as "deleted".
11
Your queued edit was based on version 5, but the server is already at version 7. How does the server notice, and what does the client do with the rejection?
The op carries the version it was edited from (If-Match or a baseVersion field), and the server compares it with the current one inside the write.
```
interface NotesApi {  @PUT("notes/{id}")  suspend fun update(    @Path("id") id: String,    @Header("Idempotency-Key") key: String,    @Header("If-Match") ifMatch: String,       // "\"5\"": the version this edit started from…
```
12
For a note title, a like counter and a shared tag list, which of last-write-wins, field-level merge and a CRDT do you pick, and what does each cost?
Choose per field, by what the value is: last-write-wins for the title, an OR-Set CRDT for the tag list, and for the counter no merge at all: send "+1 from this user" and let the server add it up.
```
// LWW: one version for the whole record, compared on a server-assigned valuefun mergeLww(local: Note, remote: Note): Note =  if (remote.serverVersion >= local.serverVersion) remote else local   // the loser's edits are gone // OR-Set: every add carries a unique tag, so a concurrent add beats a remove that never saw itdata class OrSet(…
```
13
Your merge keeps picking the wrong winner because one phone's clock is forty minutes fast. What do you order events by instead?
Order by something the server controls or by causality you can prove (a server sequence number, a Lamport clock or a version vector), and keep wall-clock time only as a display field.
```
// Lamport clock: a counter that only moves forward and catches up on every receivedata class Stamp(val counter: Long, val nodeId: String) : Comparable<Stamp> {  override fun compareTo(other: Stamp) =    compareValuesBy(this, other, Stamp::counter, Stamp::nodeId)   // node id breaks ties}…
```
14
A delta pull arrives while the user's edits to the same rows are still in the outbox. How do you apply it without wiping those edits?
Never blindly upsert a row that has pending ops: store the server row as the new base and replay the pending ops on top of it, so the screen shows server state plus the user's unsent intent.
```
suspend fun applyPage(page: DeltaPage) = db.withTransaction {  for (dto in page.rows) {    val server = dto.toEntity()    if (server.version <= (base.versionOf(server.id) ?: -1)) continue   // our own echo     base.upsert(server)                                    // the server's truth, kept apart…
```

Find the bug: a user edits a note while sync is running, and the server sometimes keeps an older title even though the note shows as synced.

suspend fun saveNote(id: String, title: String) = db.withTransaction {  notes.updateTitle(id, title, SyncState.PENDING)  val pending = outbox.pendingUpdateFor(id)  if (pending != null) outbox.updatePayload(pending.opId, title)   // coalesce: one op per note  else outbox.insert(PendingOp(opId = newUlid(), noteId = id, title = title))} suspend fun sendNext() {                     // the worker calls this in a loop  val op = outbox.next() ?: return  api.putTitle(op.opId, op.noteId, op.title)  db.withTransaction {    outbox.delete(op.opId)    notes.markSynced(op.noteId)  }}

The save merges a newer title into an op the worker is already sending; when that push succeeds, delete(op.opId) removes the row that now holds the newer title, so it is never sent and the note shows synced while the server keeps the old one.

A drain of 200 queued mutations fails on number 87. What happens to 88–200, and what changes if 87 failed with a 403 rather than a timeout?

Only 87's own chain waits; independent chains keep draining.

sealed interface Outcome {  data class Done(val entity: EntityDto) : Outcome  data object Retry : Outcome                          // keep the head, back off  data object Reauth : Outcome                         // pause the whole queue, drop nothing  data class Rebase(val server: EntityDto) : Outcome   // merge on the server state  data class Permanent(val code: Int) : Outcome…

17
You ship an update that changes the shape of a mutation payload, and thousands of users still have old-format ops sitting in their outbox. What now?
Queued mutations are persisted data written by a build that no longer exists, so the outbox needs its own version field and its own upgrade path.
```
private const val CURRENT_NOTE_UPDATE = 3private val json = Json { ignoreUnknownKeys = true }   // tomorrow's extra keys must not throw // one upgrade chain per op type, applied on readfun upgradeNoteUpdate(version: Int, raw: String): NoteUpdateV3 {  var v = version…
```
18
A post created offline has a 12 MB photo attached. How do you get the bytes up without blocking the mutation queue or filling the user's disk?
Treat the photo as its own job with its own resumable upload: copy the bytes into app storage at pick time, upload them separately in chunks, and let the post's mutation carry only a reference that becomes a server blob id.
```
private const val CHUNK = 4L * 1024 * 1024 suspend fun upload(a: Attachment): String {  val session = a.sessionUrl    ?: api.createSession(a.bytes).also { dao.setSession(a.localId, it) }…
```

Your backend sees a sync spike at 09:00 every morning and another the minute an outage ends. What is wrong with the client scheduler?

Every client retries and syncs on the same deterministic timer, so the whole fleet is phase-locked.

private val BASE = 30.secondsprivate val CAP = 6.hours // full jitter: two devices that failed together must not retry togetherfun backoffDelay(attempt: Int, retryAfter: Duration? = null): Duration {  retryAfter?.let { return it }                              // the server's number wins…

How do you test that two devices and a server actually converge, without a suite that goes red whenever CI is slow?

Replace the clock, the network and the random source with things the test controls, run both devices and a fake server in memory, and assert invariants rather than a snapshot of the final state.

class FakeServer(private val rnd: Random) {  val state = mutableMapOf<String, Note>()  val applyCount = mutableMapOf<String, Int>()      // op id -> times it really mutated state  private val seen = mutableMapOf<String, Note>()   // idempotency key -> original result  var faults: FaultPlan = FaultPlan.none…

21
Sync has been quietly broken for 2% of your users for a month. Which numbers would have told you?
The signal is the age of the oldest unsynced write, reported as a distribution per install.