Android Persistence

Pick by the shape of the data: DataStore for small settings, Room for structured data you query, app-private files for blobs and documents, MediaStore or the system picker for files the user should see, and the Keystore for the keys that encrypt the rest. DataStore (Preferences or Proto): flags and settings. Reads are a Flow, writes are suspending transactions. The modern replacement for SharedPreferences. SharedPreferences: the legacy key-value XML file, still in most codebases. Synchronous first read and no error reporting; move it with SharedPreferencesMigration. Room: SQLite with compile-checked queries, Flow results and migrations. Anything with rows, relations, search or paging. App-private files: filesDir for data you keep, cacheDir for data you can regenerate. MediaStore and the Storage Access Framework: files that should appear in the gallery or the file manager. Android Keystore: not a data store: it holds keys that never leave secure hardware, and you use them to encrypt tokens kept in one of the stores above. Two rules worth saying out loud: a growing list never goes into a key-value store, because every write rewrites the whole file; and a token never sits anywhere in plain text.

filesDir, cacheDir and noBackupFilesDir are private internal storage; getExternalFilesDir() is app-specific but on the shared volume. They differ in whether the system may delete them and whether Auto Backup copies them. filesDir: kept until uninstall and backed up; databases and SharedPreferences live beside it. cacheDir: the system or user may clear it; not backed up. Regenerable data only, such as thumbnails. noBackupFilesDir is kept, never backed up. Device-bound state: Keystore-encrypted blobs, install markers. getExternalFilesDir() and externalCacheDir: no permission, removed on uninstall, the files dir is backed up. Readable over USB, so nothing secret. "Internal" is the per-app sandbox; "external" is the shared volume, which for anything the user should see now means MediaStore or the system picker.

commit() writes to disk synchronously on the calling thread and returns true or false; apply() updates the in-memory map at once, writes to disk in the background and reports nothing. Both are thread-safe inside one process. **The apply() catch**: pending writes queue in QueuedWork, and the main thread waits for that queue when an Activity or Service stops. A burst of apply() calls can still cause an ANR. **commit() on the main thread: a disk write on the UI thread: dropped frames and a StrictMode violation. Whole-file rewrite, not multi-process**: every edit rewrites the entire XML file, and each process keeps its own copy. New code uses DataStore: asynchronous, transactional, reports errors. SharedPreferencesMigration moves existing values across once.

Room is the Jetpack layer over SQLite: you describe tables with @Entity, queries with @Dao and the database with @Database, and Room generates the code and checks your SQL at compile time. **@Entity**: a class mapped to a table; properties become columns and a @PrimaryKey is required. **@Dao**: an interface of queries: @Query with SQL, plus @Insert, @Update, @Delete and @Upsert. **@Database**: an abstract RoomDatabase that lists the entities and the schema version and exposes the DAOs. Build it once with Room.databaseBuilder and keep it a singleton. Compared with SQLiteOpenHelper and a Cursor: no hand-written mapping, a wrong column name fails the build instead of crashing a user, DAO methods can be suspend or return a Flow that re-emits on change, and migrations are structured and testable. Room is deliberately not a full ORM: you still write SQL and there is no lazy loading of related objects, so every query stays visible. Against Realm, Room is plain SQLite with standard tooling, and MongoDB deprecated Realm's device SDKs in 2024. Room's compiler runs through KSP.

getAll() is a one-shot read: it runs once in init and nothing asks again, so the insert reaches the database but not the screen. Return Flow<List<TaskEntity>> from the DAO and Room re-runs the query after every write to task. One-shot vs observable: suspend fun getAll(): List<T> answers once; fun observeAll(): Flow<List<T>> (not suspend) keeps emitting. LiveData and PagingSource return types use the same invalidation. Expose it as state: stateIn(viewModelScope, SharingStarted.WhileSubscribed(5_000), emptyList()); add() then only inserts. Do not patch it with a refresh: calling getAll() again after each insert works until another writer, such as a sync worker, forgets to. One database instance: the Flow sees only writes made through the same Room instance. A second databaseBuilder over the file is a bug; a second process needs enableMultiInstanceInvalidation(). SharedPreferences equivalent: wrap OnSharedPreferenceChangeListener in callbackFlow; DataStore's data is already a Flow.

DataStore fixes what SharedPreferences gets wrong: reads are a Flow, writes are suspending atomic transactions, and failures arrive as exceptions instead of being dropped. Preferences DataStore is schemaless key-value; Proto DataStore stores one typed object. What SharedPreferences gets wrong: synchronous getters that can block the UI thread, apply() that swallows errors and can ANR on lifecycle transitions, no read-modify-write transaction, and no change stream without a listener. Preferences DataStore: typed keys (stringPreferencesKey) over an untyped file, so a typo in a key name silently returns the default. The easiest move from SharedPreferences, with SharedPreferencesMigration. Proto DataStore: one object with a schema: a protobuf message or any type with a Serializer. A @Serializable data class with kotlinx.serialization works and skips protoc. Typed defaults and safe evolution; best for settings that belong together. Shared rules: exactly one DataStore instance per file per process (a second one throws), edit {} and updateData {} are atomic read-modify-write, and each write replaces the whole file through a temp file and a rename. Not a database: no queries and no partial updates; a list that grows belongs in Room.

runBlocking makes the main thread wait for a disk read and a parse during startup, which is exactly the blocking DataStore was built to remove and a source of startup jank or ANRs. Treat the theme as state that is not known yet, and keep the splash screen up until it arrives. Second bug: the value is read once. A theme changed later in settings will not apply until the next launch; collect data as a Flow instead. The fix: the ViewModel exposes StateFlow<String?> that starts as null, installSplashScreen().setKeepOnScreenCondition { ... } holds the first frame until it is not null, and the UI collects it with collectAsStateWithLifecycle(). Handle errors in the Flow: catch { if (it is IOException) emit(emptyPreferences()) else throw it }; otherwise one unreadable file crashes every collector. **When first() is fine**: in a coroutine off the main thread, such as a Worker that needs the current value once.

Every SettingsRepository builds its own DataStore over settings.preferences_pb. The first one is never closed, so when the next ViewModel's instance reads the same file DataStore throws IllegalStateException on that first read or write (not at construction): "There are multiple DataStores active for the same file". Why the first instance stays alive: PreferenceDataStoreFactory.create runs in its own CoroutineScope(Dispatchers.IO + SupervisorJob()) by default. Clearing the ViewModel drops the reference, but nothing cancels that scope, so the file stays registered as active for the rest of the process. Why DataStore refuses: each instance caches the file in memory and serialises only its own writes. Two instances over one file would overwrite each other's edits and serve stale values, so the check turns silent data loss into a crash. How the fix works: one instance per file per process: the top-level preferencesDataStore delegate caches it and uses applicationContext. With Hilt, a @Singleton provider hands repositories a DataStore<Preferences> instead. Several processes: a singleton is per process; a file shared with a :sync process needs MultiProcessDataStoreFactory. Catching it next time: create DataStore only in the DI module or a top-level delegate and flag PreferenceDataStoreFactory.create anywhere else in review; a UI test that opens the screen twice reproduces the crash.

A query is disk I/O that can take tens of milliseconds, so Room throws IllegalStateException: Cannot access database on the main thread for blocking DAO calls. suspend and Flow DAO methods are safe to call from the main thread because Room moves the work to its own executors. Blocking DAO methods: a non-suspend function returning List<T> runs on the caller's thread, which is the case the check guards. allowMainThreadQueries() turns the check off: acceptable in tests, never in production. **suspend methods**: Room runs them on its query or transaction executor and resumes your coroutine afterwards, so wrapping them in withContext(Dispatchers.IO) is redundant. **Flow queries**: not suspend; the query runs on Room's executor when collected and again after every invalidation, so flowOn(Dispatchers.IO) adds nothing. **withTransaction { }: takes one thread from the transaction executor and confines the block to it, so suspend DAO calls inside join the same transaction. It commits when the block returns and rolls back on an exception or cancellation. The classic hang**: runBlocking inside withTransaction blocks the transaction thread that the inner DAO call needs, and a DAO call from a coroutine launched outside the block (a new scope) waits for the writer lock the transaction holds. Inside the block call DAO functions directly, and make network calls before it.

SQLite stores only integers, reals, text and blobs, so for other types such as Instant or a list you write a @TypeConverter pair that maps the value to a supported type and back, and register it with @TypeConverters. Two functions: one per direction, in a class or an object. Use @ProvidedTypeConverter with addTypeConverter(...) when the converter needs a dependency. Scope: register on @Database to cover everything, or on an entity, DAO or field to keep it narrow. Enums: since Room 2.3 Room stores enums by name without a converter. If you write your own, never store the ordinal: reordering the constants silently remaps every existing row. The main limitation: a converted column is opaque to SQL. You cannot filter, sort or index by an element of a list serialised to JSON, so anything you query belongs in its own column, an @Embedded class or a child table. Cost: the converter runs for every row read, so a large object serialised into one column is parsed in full on every query.

You declare the shape with @Embedded for the parent and @Relation for the children, and Room runs one query per side and joins the results in memory. Because that is several queries, the DAO method needs @Transaction, or a write between them can produce a graph that never existed. One-to-many: a plain data class (not an @Entity) with @Embedded val author and @Relation(parentColumn = "id", entityColumn = "authorId") val books: List<BookEntity>. Many-to-many: a junction entity with a composite primary key, referenced through associateBy = Junction(...). Integrity: @ForeignKey(onDelete = CASCADE) stops orphans, and the child column needs an @Index, or every parent lookup scans the table; Room warns about both. Loads everything: a relation fetches all children. An author with ten thousand books loads ten thousand rows, so large child lists need paging or their own query. No filtering on the relation: Room generates the child query. "Authors with at least one unread book" means writing the JOIN yourself, or a @Query returning a multimap such as Map<AuthorEntity, List<BookEntity>>. **@Transaction beyond relations**: any DAO method that runs several statements needs it, so a partial write cannot survive a failure.

REPLACE deletes the conflicting row and inserts a new one; it is not an update. So cascading deletes fire, child rows disappear, and columns you did not set fall back to their defaults. @Upsert tries the insert and, on conflict, updates the existing row in place. **ABORT: the default. The insert throws a constraint exception and the transaction rolls back. IGNORE**: the conflicting row is skipped silently and @Insert returns -1 for it. The source of "the sync ran and nothing changed". **REPLACE**: DELETE then INSERT: ON DELETE CASCADE children are destroyed, an autoGenerate key gets a new value, and every column not in the new object resets. **@Update: changes an existing row by primary key and does nothing when the row is absent. @Upsert (Room 2.5+) inserts or updates in place: nothing is deleted and children survive. Return values**: @Insert returns the new rowid (or -1 per skipped row), @Update and @Delete return the number of affected rows, which tells "created" from "already there" without a second query. For synced data make the server id the @PrimaryKey (or look up the local id by remoteId before the upsert), because @Upsert resolves a conflict only through the primary key; a unique index on another column makes the insert fail and the follow-up update by id match nothing. Local-only columns such as a favourite flag need an explicit UPDATE of the server fields, because an upsert of a fresh object overwrites them too.

Room stores the schema version and a schema hash in the database file. On open it runs migrations up to the current version, then compares the hash with the compiled schema; this crash means they differ: the schema changed without a version bump, or a migration built a slightly different table. Every schema change: bump version and give Room a path: a Migration(from, to) with SQL, or @AutoMigration(from, to) that diffs the exported schemas. The other crash: "A migration from 4 to 5 was required but not found" means you bumped the version and supplied no path. Export schemas: exportSchema = true plus room { schemaDirectory("$projectDir/schemas") } from the Room Gradle plugin; commit the JSON. Auto-migrations and migration tests need those files. Ambiguous changes: a renamed or deleted column fails the build until you give the auto-migration a spec with @RenameColumn or @DeleteColumn. Hand-written SQL: a new NOT NULL column needs a DEFAULT; changing a type or constraint means create a new table, copy rows, drop, rename. Every column, default and index must match the entity exactly. **fallbackToDestructiveMigration(dropAllTables = true): deletes the user's data. Only for a pure cache. Never edit a released migration**: a user who skipped releases runs 3 to 4 to 5, so every link must keep working; test the chain with MigrationTestHelper.

Test a DAO against a real in-memory database rather than a mock, because the value of Room is the SQL it generates. Test a migration with MigrationTestHelper, which builds the old schema from the exported JSON, runs your migration and validates the result. DAO tests Room.inMemoryDatabaseBuilder(...) gives a fresh database per test that never touches disk; close it in teardown. They need real SQLite, so they run as instrumented tests or under Robolectric. Assert behaviour, not calls: the filter filters, the conflict strategy does what you assumed, a Flow re-emits after a write (Turbine helps), a cascade removes children. allowMainThreadQueries() is fine here and nowhere else. Migration tests createDatabase(name, oldVersion) builds the old schema; insert rows with raw SQL, because the old entity classes no longer exist. runMigrationsAndValidate(name, newVersion, true, MIGRATION_2_3) checks the schema against the exported JSON. Then assert the data survived. A migration that recreates a table and forgets to copy rows passes schema validation. Also migrate from version 1 all the way up, as a user who skipped releases does. Keep repository tests above the DAO on a fake; database tests exist to check the SQL.

Each upsert outside a transaction is its own transaction, and Room's invalidation is per table: every commit tells every observer of task to re-run its whole query. Three hundred rows can mean up to three hundred re-queries and a burst of emissions. What the tracker does Room adds triggers to observed tables that record "this table changed" in an internal log table. After each transaction it notifies the observers of the changed tables. It never knows which rows changed, so the query re-runs in full. A query over several tables, or a @DatabaseView, is invalidated by a write to any of them. The fix Wrap the batch in db.withTransaction { } or a @Transaction DAO method: one commit, one notification, one re-query, and no observer sees a half-applied sync. Pass the whole list to one @Upsert call, which also runs in a single transaction. distinctUntilChanged() drops identical emissions, but the query has already run; it saves recomposition, not I/O. Observe narrow queries. A LIMIT 50 query is invalidated as often as SELECT *, but costs far less to re-run.

addOne() reads the quantity and writes current + 1 as two separate statements, so two taps can both read 3 and both write 4: one increment is lost (a lost update). Do the read and the write in one transaction, or let SQL increment the column itself. Why it happens even on the main thread: every suspend DAO call hops to Room's executor and suspends the caller, so the coroutine launched by the next tap runs its find() before the first one reaches upsert(). A sync worker writing the same row makes it worse. How the fix works: withTransaction runs the block as one write transaction on Room's single writer connection; a second addOne() waits until the first commits and then reads the new value. Simpler where possible: UPDATE cart_item SET quantity = quantity + 1 WHERE productId = :id is atomic on its own. Read-then-write needs a transaction only when the new value is computed in Kotlin. Weaker fixes: a Mutex in the repository serialises only this repository's callers, not a sync worker or other DAO paths that write the table; @Upsert alone fixes nothing because it writes a value already computed from a stale read. Catching it next time: a test that launches 100 addOne() calls concurrently against an in-memory database and expects 100; in review, flag any DAO read whose result feeds a DAO write outside a transaction.

Generate an AES key in the Android Keystore, encrypt the token with it and store only the ciphertext in DataStore, a file or Room. EncryptedSharedPreferences did this wrapping for you, but androidx.security:security-crypto is deprecated. Android Keystore: keys are generated inside the TEE or StrongBox, cannot be exported, can require user authentication, and can be invalidated when a new biometric is enrolled. **After security-crypto: existing data keeps working, but new code uses its own small Keystore wrapper (below) or Tink. Plan a migration: read with the old API once, write with the new one. Whole database**: SQLCipher (net.zetetic:sqlcipher-android with SupportOpenHelperFactory) encrypts the Room file; keep its passphrase wrapped by a Keystore key. It costs a native library in the APK and some query speed. Good practice: keep the access token in memory and only the refresh token at rest; never store passwords; gate sensitive actions with BiometricPrompt and a CryptoObject, so authentication actually unlocks the key; exclude encrypted files from backup.

Since Android 10 and 11, apps cannot write arbitrary paths on shared storage; they get private directories, the MediaStore collections and the system file picker. Creating a file through MediaStore or the picker needs no storage permission. App-private directories: filesDir, cacheDir, getExternalFilesDir(). No permission, closed to other apps on Android 11+ (getExternalFilesDir() is still visible over USB), deleted on uninstall. MediaStore: shared collections for images, video, audio and downloads. Insert a row, get a content URI, write to its stream. Adding files and reading your own needs no permission; reading other apps' media needs the media permissions or the Photo Picker. Storage Access Framework: ActivityResultContracts.CreateDocument and OpenDocument. The user picks the location, you get a URI for exactly that file, and cloud providers work too. For an export: the picker is usually right, because the file lands where the user chose; MediaStore Downloads suits filing it automatically with no dialog. Details: mark a MediaStore entry IS_PENDING while writing; call takePersistableUriPermission to keep a picked URI after restart; write streams off the main thread. **MANAGE_EXTERNAL_STORAGE**: restricted by Play policy to file managers and similar apps; not the answer here.

FileOutputStream(target) truncates the file before the first byte is written, so a crash mid-write leaves half a file and no old copy. Write to a temp file in the same directory, sync and rename it over the target, or use AtomicFile, which keeps a backup until the write completes. If you do it by hand: same directory for the temp file: a rename is atomic only within one filesystem, so readers see the old file or the new one, never half of either. **If you do it by hand: fd.sync() before the rename: the rename orders metadata, not your bytes; without the sync a power loss can leave an empty file under a valid name. AtomicFile**: startWrite() moves the old file to a backup, finishWrite(stream) syncs and deletes it, failWrite(stream) restores it; readFully() also recovers from a leftover backup. Right directory: data you must keep goes in filesDir, never in cacheDir, which the system can empty at any time. Threading: file I/O belongs on Dispatchers.IO, and one writer per file; concurrent writers still race. DataStore uses the same temp-file-and-rename approach internally, which is why it does not tear files.

The UI observes only Room: the network writes responses into Room and the Room Flow delivers them, so the screen shows cached data instantly, works offline and has a single update path. Read path: the repository returns dao.observe...() as a Flow; a separate refresh() calls the API and writes the result in one transaction. When to refresh: on screen open, on pull-to-refresh, or when data is stale: keep a fetchedAt per key and skip the request while it is fresh. OkHttp's HTTP cache can sit underneath, but it does not replace a queryable local copy. Errors: a failed refresh keeps showing cached rows plus a non-blocking error; it never empties the screen. Applying a response: delete rows missing from a full list and upsert the rest inside withTransaction, so observers get one consistent update. Keep local-only columns out of the overwrite. Large lists: Paging 3 with a Room PagingSource and a RemoteMediator applies the same idea page by page. Local edits: write to Room first and mark them pending; a WorkManager job pushes them later. Conflict rules are a separate design decision.

Clearing the token only stops new requests being authorised; every local copy of the previous user's data stays on the device and the next person to sign in will see it. Stop the writers first, then wipe every user-scoped store, then go to the login screen. Background work: cancel sync workers first, or one wakes later and uploads the old user's pending changes with the new user's token. Push registration: unregister the device token on the server while you still have the session. Credentials: at rest and in memory; an interceptor holding the old token in a field keeps sending it. Database: clearAllTables(); deleting the file under open connections crashes instead. DataStore and SharedPreferences: user-scoped keys only. Keeping device-scoped ones such as theme or onboarding-seen should be a deliberate choice. Caches: the OkHttp cache, the image loader's memory and disk caches, exported files in cacheDir. Put all of this in one session-clearing component that every store registers with, so a feature added next quarter cannot leak. A database file per account makes isolation structural. Test it by signing in as one user, logging out and signing in as another, also offline.

In WAL mode SQLite appends committed changes to a separate -wal file instead of rewriting database pages in place, so readers keep reading the last committed state while a writer writes. Room enables it by default on most devices, so reads and writes on different threads do not block each other. How it works A commit appends pages to app.db-wal; app.db-shm is the shared-memory index into that log. A checkpoint copies WAL pages back into the main file, automatically once the log passes a size threshold. Readers do not block the writer and the writer does not block readers. Writers are still serialised: one at a time. What it means in Room JournalMode.AUTOMATIC, the default, picks WAL unless the device is low-RAM; setJournalMode(JournalMode.TRUNCATE) opts out. Room keeps several reader connections and one writer, so a long write transaction does not freeze observed queries, but it does queue every other write. The three files are one database. Copying, sharing or backing up only app.db can lose committed data; checkpoint first (PRAGMA wal_checkpoint(FULL)) or close the database before copying. WAL works across processes, but Room's change notifications still need enableMultiInstanceInvalidation(). Trade-offs Two extra files and a WAL that can grow large under a long-running read that blocks checkpoints. Slightly slower reads while the log is big; in exchange, much better concurrency and fewer fsyncs per commit.

An index is a separate B-tree sorted by the indexed columns, so SQLite can find matching rows in logarithmic time instead of reading the whole table. A query that got slow as data grew is nearly always a full table scan, and EXPLAIN QUERY PLAN shows it. Finding it Run the query through EXPLAIN QUERY PLAN from a @RawQuery in a test or in the Database Inspector. SCAN means every row was read; SEARCH ... USING INDEX means the index was used. Measure on a device with realistic data: everything is fast at a hundred rows, which is why the problem shipped. Fixing it Index the columns you filter, join and sort on. A composite index serves queries that use its columns left to right: ["chatId", "createdAt"] covers "this chat, newest first". Do not index everything; each index costs write time and file size. Select only the columns the screen draws, into a small data class, instead of loading body blobs per row. Count, filter and sort in SQL, not in Kotlin after loading rows. Never filter on a @TypeConverter column; promote the field to a real column. Prefer keyset pagination on (createdAt, id) (WHERE createdAt < :ts OR (createdAt = :ts AND id < :id), LIMIT n) over a large OFFSET, which walks every skipped row; the id tie-break keeps rows with the same createdAt. LIKE '%term%' cannot use an index; use an FTS table (@Fts4). Batch writes in one transaction instead of a commit per row. Then re-read the query plan and re-measure on the same dataset.

Auto Backup is on by default: it copies filesDir, databases and SharedPreferences to the user's Google account and restores them on reinstall or on a new phone. Keystore keys are never backed up, so anything you encrypted comes back as ciphertext with no key to open it. What is backed up With the default android:allowBackup="true": filesDir, databases/, shared_prefs/ and getExternalFilesDir(), up to a per-app quota. cacheDir and noBackupFilesDir are never included. From API 31 you configure it with android:dataExtractionRules, with separate <cloud-backup> and <device-transfer> sections; android:fullBackupContent covers older versions. For apps targeting Android 12+, allowBackup="false" stops cloud backup but not device-to-device transfer; exclude files in the rules for that. What breaks Keystore-encrypted data: the bytes return, the key does not, and the first decrypt throws instead of returning nothing. Flags restored from another device, such as onboardingSeen, no longer match the new device's state. A restored install can show the previous device's cached rows before anyone has signed in. What to ship Keep Keystore-encrypted files in noBackupFilesDir or exclude them in the rules. Still catch the decryption failure, wipe that store and send the user to sign in, because older backups already exist. Ship the marker one release before you act on it, or create it on first launch when a valid session exists; otherwise every existing install looks like a restore. Detect a restore with a marker file in noBackupFilesDir: a database present without the marker came from another device.