Irbisa · cheatsheetSeptember 13, 2026

Offline Sync & Conflicts

Mutation outbox, idempotency keys, delta sync, tombstones, LWW vs CRDT, backoff, convergence

Senior Developer16 itemscompressed for a skim

A mutation has sat in the outbox for three weeks across two app updates. What must that row have stored for it to still be replayable?

Medium

Everything needed to rebuild the request from data alone — an outbox row is a durable record of intent, not a paused function call.

@Entity(tableName = "outbox")
data class PendingOp(
  @PrimaryKey val opId: String,      // idempotency key: minted here, never regenerated
  val groupKey: String,              // aggregate root — the ordering scope
  val seq: Long,                     // monotonic within the group
  val entityType: String,
…

02
What does a locally created row carry so the list can show sending, failed and rolled back without the row flickering or jumping?
Medium
The row stores its own sync state as a column, so the badge is a pure function of the database and never of what the network happens to be doing at that moment.
```
enum class SyncState { SYNCED, PENDING, SENDING, FAILED }

@Entity(tableName = "notes")
data class NoteEntity(
  @PrimaryKey val localId: String,      // stable for the row's whole life
  val remoteId: String? = null,         // arrives later, never becomes the key
…
```
03
A payment POST times out before the client sees a response, so the client retries. What keeps that from charging twice, and who mints the key?
Medium
The client mints an idempotency key once, when the mutation is enqueued, and every attempt sends that same key; the server stores the key together with the first execution's result and replays that stored response instead of running the write again.
```
interface OrdersApi {
  @POST("orders")
  suspend fun create(
    @Header("Idempotency-Key") key: String,
    @Body body: CreateOrder
  ): Response<Order>
…
```
04
You create a row offline with a client UUID and the server returns its own id on first sync. How do you reconcile them without breaking every reference?
Medium
Keep the client id as the primary key for the row's whole life and store the server id beside it in a nullable column — rewriting the primary key is what breaks foreign keys, queued payloads and list identity.
```
@Entity(tableName = "posts")
data class PostEntity(
  @PrimaryKey val localId: String,   // ULID minted on the device, never rewritten
  val remoteId: String? = null,      // filled in on the first successful sync
  val title: String
)
…
```
05
A queued comment refers to a post that is itself still sitting unsynced in the outbox. How does the queue guarantee the post goes first?
Medium
Order the queue per dependency group rather than globally: every op carries a groupKey for the aggregate it belongs to and a monotonic seq inside it, so the post and its comment drain in order while unrelated groups drain in parallel.
```
@Dao
interface OutboxDao {
  @Query("SELECT DISTINCT groupKey FROM outbox WHERE status = 'READY' AND nextAttemptAt <= :now")
  suspend fun readyGroups(now: Long): List<String>

  // Strictly one op per group in flight, in seq order
…
```
06
Your pull sends ?updated_since=<last timestamp> and users report rows that simply never arrive. What must the server guarantee for that cursor to be correct?
Medium
A timestamp cursor is only correct if the server filters and orders on the same key, breaks ties deterministically, and stamps rows at commit time — otherwise page boundaries and concurrent transactions both leave permanent holes.
```
@Serializable
data class Cursor(val updatedAt: Long, val id: String)   // composite: value + tie-break

@Serializable
data class DeltaPage(val rows: List<NoteDto>, val next: Cursor?, val hasMore: Boolean)
…
```
07
A phone comes back after two months offline and still shows notes that everyone else deleted. What did the sync protocol get wrong?
Medium
Absence from a delta response means "unchanged", not "deleted" — deletes must travel as explicit tombstones, and a client whose last sync predates the tombstone retention window must not be allowed to do a delta pull at all.
```
@Serializable
data class NoteDto(
  val id: String,
  val updatedAt: Long,
  val deleted: Boolean = false,      // the tombstone marker, carried in the delta
  val body: String? = null           // absent for a tombstone: deleted content is not shipped
…
```
08
For a note title, a like counter and a shared tag list, which of last-write-wins, field-level merge and a CRDT do you pick, and what does each cost?
Medium
Choose per field, by what the value actually is: last-write-wins for scalars nobody co-edits, field-level merge when two devices touch different fields of one record, and a CRDT only where the same value is genuinely mutated concurrently.
```
// LWW: one version for the whole record, compared on a server-assigned value
fun mergeLww(local: Note, remote: Note): Note =
  if (remote.serverVersion >= local.serverVersion) remote else local  // the loser's edits are gone

// Field level: a version per field, merged column by column
data class Versioned<T>(val value: T, val version: Long)
…
```

Your merge keeps picking the wrong winner because one phone's clock is forty minutes fast. What do you order events by instead?

Hard

Order by something the server controls or by causality you can prove — a monotonic server sequence, a Lamport clock, or a version vector — and demote wall-clock time to a display field.

// Hybrid logical clock: sorts like a timestamp, ordered like a Lamport clock
data class Hlc(val millis: Long, val counter: Int, val nodeId: String) : Comparable<Hlc> {
  override fun compareTo(other: Hlc) =
    compareValuesBy(this, other, Hlc::millis, Hlc::counter, Hlc::nodeId)
  fun encode() = "%013d:%05d:%s".format(millis, counter, nodeId)
}
…

10
A drain of 200 queued mutations fails on number 87. What happens to 88 through 200?
Hard
It depends entirely on whether they depend on 87: the queue must stall the chain that contains the failure and keep draining everything independent of it.
```
@Entity(tableName = "outbox")
data class Op(
  @PrimaryKey val id: String,        // also the idempotency key
  val chainKey: String,              // entity id: order is promised only inside a chain
  val seq: Long,
  val type: String,
…
```

A queued edit comes back 403 because the user lost access to that project. How do you get it out of the queue without wedging sync, and what does the user see?

Hard

Classify the response first: a status that will never succeed has to leave the queue in one transaction that also reverts the local state and records evidence the user can act on.

sealed interface Outcome {
  data class Done(val entity: EntityDto) : Outcome
  data object Retry : Outcome                          // I/O, 408, 425, 429, 5xx
  data object Reauth : Outcome                         // 401 — pause, do not drop
  data class Rebase(val server: EntityDto) : Outcome   // 409
  data class Permanent(val code: Int, val message: String?) : Outcome
…

12
You ship an update that changes the shape of a mutation payload, and thousands of users still have old-format ops sitting in their outbox. What now?
Hard
Queued mutations are persisted data written by a build that no longer exists, so the outbox needs its own version field and its own upgrade path — a Room migration moves the table, not the meaning of the blobs inside it.
```
private const val CURRENT_NOTE_UPDATE = 3

// Self-describing row: stable type name, payload version, opaque body
@Entity(tableName = "outbox")
data class OpRow(
  @PrimaryKey val id: String,
…
```
13
A post created offline has a 12 MB photo attached. How do you get the bytes up without blocking the mutation queue or filling the user's disk?
Hard
Treat the file as its own job with its own resumable lifecycle: copy the bytes into app storage at capture time, upload them separately, and let the post's mutation carry only a reference that resolves to a server blob id.
```
@Entity(tableName = "attachments")
data class Attachment(
  @PrimaryKey val localId: String,
  val path: String,              // OUR copy — the picker's content URI is revoked, the file is not
  val bytes: Long,
  val sessionUrl: String? = null,
…
```

Your backend sees a sync spike at 09:00 every morning and another the minute an outage ends. What is wrong with the client scheduler?

Hard

Every client is retrying on the same deterministic timer, so the whole fleet is phase-locked — the fix is randomization, not a different fixed delay.

private val BASE = 30.seconds
private val CAP = 6.hours

// Full jitter: two devices that failed together must not retry together
fun backoffDelay(attempt: Int, retryAfter: Duration? = null): Duration {
  retryAfter?.let { return it }                              // the server's number wins
…

How do you test that two devices and a server actually converge, without a suite that goes red whenever CI is slow?

Hard

Replace the clock, the network and the random source with things the test controls, run both devices and a fake server in memory, and assert invariants rather than a snapshot.

class FakeServer(private val rnd: Random) {
  val state = mutableMapOf<String, Note>()
  val applyCount = mutableMapOf<String, Int>()      // op id -> times it really mutated state
  private val seen = mutableMapOf<String, Note>()   // idempotency key -> original result
  var faults: FaultPlan = FaultPlan.none
…

16
Sync has been quietly broken for 2% of your users for a month. Which numbers would have told you, and what may you log about a failed mutation?
Hard
The age of the oldest unsynced write, reported as a distribution per install, is the signal — a request success rate cannot see this cohort, because a wedged client eventually stops sending requests at all.
```
// Sampled on every foreground — a distribution, not an average
data class SyncHealth(
  val pendingOps: Int,
  val oldestPendingAgeMs: Long,   // the number that exposes a stuck cohort
  val deadLettered: Int,
  val sinceLastSuccessMs: Long,
…
```

A mutation has sat in the outbox for three weeks across two app updates. What must that row have stored for it to still be replayable?

What does a locally created row carry so the list can show sending, failed and rolled back without the row flickering or jumping?

A payment POST times out before the client sees a response, so the client retries. What keeps that from charging twice, and who mints the key?

You create a row offline with a client UUID and the server returns its own id on first sync. How do you reconcile them without breaking every reference?

A queued comment refers to a post that is itself still sitting unsynced in the outbox. How does the queue guarantee the post goes first?

Your pull sends ?updated_since=<last timestamp> and users report rows that simply never arrive. What must the server guarantee for that cursor to be correct?

A phone comes back after two months offline and still shows notes that everyone else deleted. What did the sync protocol get wrong?

For a note title, a like counter and a shared tag list, which of last-write-wins, field-level merge and a CRDT do you pick, and what does each cost?

Your merge keeps picking the wrong winner because one phone's clock is forty minutes fast. What do you order events by instead?

A drain of 200 queued mutations fails on number 87. What happens to 88 through 200?

A queued edit comes back 403 because the user lost access to that project. How do you get it out of the queue without wedging sync, and what does the user see?

You ship an update that changes the shape of a mutation payload, and thousands of users still have old-format ops sitting in their outbox. What now?

A post created offline has a 12 MB photo attached. How do you get the bytes up without blocking the mutation queue or filling the user's disk?

Your backend sees a sync spike at 09:00 every morning and another the minute an outage ends. What is wrong with the client scheduler?

How do you test that two devices and a server actually converge, without a suite that goes red whenever CI is slow?

Sync has been quietly broken for 2% of your users for a month. Which numbers would have told you, and what may you log about a failed mutation?

Your pull sends `?updated_since=<last timestamp>` and users report rows that simply never arrive. What must the server guarantee for that cursor to be correct?