AI Features in Mobile Apps

Default to the server path, because it works on every device, and use the on-device model only for bounded, short or private tasks, and only when the availability check passes. Local is free, private and offline, but the model is small and only recent hardware has one. iOS 26+: Foundation Models gives you Apple's model through LanguageModelSession, and SystemLanguageModel.default.availability says whether this phone can run it Android: ML Kit GenAI runs Gemini Nano through AICore: task APIs (summarize, proofread, rewrite, image description) plus a Prompt API for free-form prompts, on recent flagships only; other models mean shipping your own weights with LiteRT / LiteRT-LM When to use which: bounded task, short input, private data: local first; quality is the product: server, with local as the offline fallback Common mistake: treating on-device as a capability rather than a per-call probe; most of your installed base will fail the availability check for a while yet

Every turn re-sends and re-bills all earlier turns, so input tokens grow each turn, total spend grows roughly quadratically, time to first token creeps up, and at some point the request overflows the context window and fails. Budget the window in tokens, not messages. Tokens, not characters: models bill and limit in tokens: roughly 4 characters per token for English, noticeably more tokens per word for Russian; use the usage numbers the API returns, not text.length Budget the window: fixed system prompt, then the newest turns that fit the budget, leaving room for the answer (max_output_tokens) Summarise the tail: fold older turns into a running summary, so a long chat keeps its context at a fixed cost Prompt caching: keep the stable prefix byte-identical so the vendor's cache discounts it; a timestamp at the top kills the cache Hidden race: two quick sends both append to history, and replies land in the wrong order; serialize sends or key turns by id Where it lives: the server assembles the window from stored turns; the client sends a turn id, not the history

It never calls awaitClose, so callbackFlow fails at runtime with IllegalStateException and nothing cancels the EventSource; it also never completes the flow on a normal end of stream and can silently drop tokens when the buffer fills. **awaitClose { source.cancel() }: keeps the producer alive and closes the socket when the collector is cancelled; that close is what lets the server stop generating onClosed → close()**: without it a finished answer never completes the flow, and the UI spins forever; many APIs also send a [DONE] sentinel HTTP errors: OkHttp calls onFailure with t == null and the Response for a non-2xx status; turn that into an exception, or a 429 looks like a hang Dropped tokens: trySend into the default 64-slot buffer fails when the collector is slow; use buffer(Channel.UNLIMITED) (text is small) or trySendBlocking Timeouts: use client.newBuilder().readTimeout(0, TimeUnit.SECONDS).build() for streams; the default 10 s read timeout cuts a slow generation Alternative: Ktor's SSE plugin gives you client.sse { incoming.collect { } } with cancellation built in

Each network chunk is decoded on its own, so a Cyrillic letter or an emoji whose UTF-8 bytes straddle two chunks is cut in half, and both halves turn into �. ASCII is one byte per character, which is why English never breaks. Why: a Cyrillic letter takes 2 bytes in UTF-8, an emoji 4; TCP and chunked encoding cut the stream at arbitrary byte offsets, not at character boundaries, and String(bytes, UTF_8) replaces an incomplete sequence with U+FFFD instead of waiting for the rest The fix: one InputStreamReader (reader(Charsets.UTF_8)) for the whole stream: its decoder holds the leading bytes of a split character until the last byte arrives, and read(CharArray) returns as soon as some whole characters are ready, so the text still streams Already safe: line readers such as OkHttp's BufferedSource.readUtf8Line() (behind the SSE adapter) and Ktor's readUTF8Line() decode whole lines; the bug shows up with raw byte reads: plain-text or NDJSON streams, a hand-written parser Catch it next time: a unit test that feeds a Russian answer with an emoji through a fake stream returning 1–3 bytes per read; English-only fixtures never hit it

One token is not one UI update. Buffer the deltas, flush them on a frame cadence, append into a single text node, and auto-scroll only while the user is already at the bottom. The stream has a data rate, the screen has a frame rate. Convert between them once, at the boundary. Why per-token rebuilds hurt: laying out a growing paragraph redoes the whole paragraph, and re-parsing Markdown on every delta doubles it Throttle: accumulate off the main thread and emit at most every 30–60 ms; nobody reads faster than that Stable identity: the streaming answer is one list item with a stable key, not one item per token Scrolling: a chat list uses reverseLayout = true with the newest message as item 0: the list is anchored to the bottom, so the growing answer stays in view while the user is there, and older messages stay put once they scroll up; show a jump-to-latest button when firstVisibleItemIndex > 0

The cancellation must reach the socket: closing the response body is what stops the meter. And the partial text already on screen is state, so save it before the scope dies. Right scope: the call belongs to the scope that owns the screen; a generation launched on a process-wide scope keeps streaming and billing after the screen is gone Closing is the signal: cancelling the coroutine suspended on the read closes the connection (on iOS, cancelling the Task), provided the stream adapter cancels the call on cancellation (awaitClose { source.cancel() }, Ktor); checking a flag stops nothing **Save inside withContext(NonCancellable)**: a cancelled coroutine cannot suspend, so a plain save() in finally is silently skipped Not an error: show the partial text with a retry, not a red banner What the server needs: a proxy that turns a closed client connection into an abort upstream, a generation id to target, and its own token cap and idle timeout: process death runs no finally at all

Time to first token is the number that matters: about 100 ms feels instant, a second is fine if the tap was acknowledged, and after a few seconds people leave. Once tokens are moving, total duration barely matters, because reading is slower than generation. Honest tricks remove real waiting or make it visible; theatre fakes work that is not happening. Measure it in parts: split the wait into your own work, your proxy and the vendor's queue; track it at p95 on real networks, not on office wifi Honest: start the request on intent rather than on submit, cache the fixed system prefix on the server, keep an on-device session warm, and draw the skeleton with a working Cancel at once Theatre: a typewriter animation over an answer you already have, deliberately slowed streaming, or a determinate progress bar for a generation whose length you do not know

Load the .tflite file memory-mapped into one Interpreter, reuse it, and run it off the main thread. A delegate moves supported ops from the CPU to the GPU or NPU: faster and lighter on battery for large models, at the cost of init time and silent per-op fallback. LiteRT: the 2024 rename of TensorFlow Lite; the runtime comes bundled or from Google Play services, and the org.tensorflow.lite API is unchanged Create once: building an interpreter costs tens to hundreds of ms and a lot of memory; it is not thread-safe, so use it from one thread and close() it, and its delegate, when the feature goes away Check before delegating: CompatibilityList says whether the GPU delegate works on this device; unsupported ops fall back to the CPU, and every CPU↔GPU hop copies tensors, so measure with the benchmark tool NNAPI: deprecated since Android 15; NPUs are reached through LiteRT's newer accelerator APIs instead Size: quantize to int8 or fp16 to cut the file 2–4x; for LLMs use LiteRT-LM (MediaPipe LLM Inference is in maintenance mode) rather than a raw interpreter

Both treat generated output as your content: you must prevent and moderate harmful output and let users report it inside the app, and you must disclose and get consent before personal data goes to a third-party AI provider. Play AI-Generated Content policy: the app must prevent restricted content, and users must be able to report or flag offensive output in the app, without leaving it Play Data safety: prompts, photos and voice sent to a model vendor are data collected and shared; declare them, including what the vendor's SDK collects App Store 5.1.2(i): since the November 2025 update, the app must clearly say when personal data is shared with a third-party AI and get explicit permission first Generated content shown to others: falls under both stores' user-generated content rules: filtering, reporting, blocking In practice: a consent screen before the first cloud call, a report action on every answer, server-side moderation of input and output, and a privacy policy that names the vendor

The model never executes anything: it returns a tool name with JSON arguments, your code validates and runs it, and you send the result back so the model can continue. The app decides what is callable and what needs the user's confirmation. The loop: declare tools with a name, description and JSON schema; the model answers with a call instead of text; you run it and return the result as a tool message; repeat until it answers in text, with a cap on rounds Where tools run: search and your own database belong on the server; calendar, contacts, location and the current screen exist only on the phone, so the server hands those calls to the client and waits Arguments are model output: decode into a typed class, check ranges and permissions, and answer unknown tool names with an error, never a guess Side effects need a human: read-only tools run automatically; anything that sends, pays, deletes or books shows the exact arguments for confirmation and carries an idempotency key Runtime permissions: ask at the moment the tool needs one, not at app start

A vendor key inside the APK is public: anyone can extract it. But a proxy that only hides the key and forwards whatever body the client sends is still an open inference endpoint billed to you, so the server must own the prompt, the model and each user's budget. The rule: the client says what the user wants; the server decides what the model is asked and how much this user gets. Own the prompt: the client posts intent and inputs (thread id, tone, locale), never a prompt string Own the model: the model id, decoding parameters and token ceiling live on the server Quota per user: a daily ceiling and a 429 the app can render; on mobile an IP is not an identity, one carrier NAT is thousands of people Attestation: Play Integrity or App Attest on the token-issuing path, plus your own per-request token accounting Still pass through: the stream, cancellation down to the upstream call, and a request id in the response headers

Treat everything the model reads as untrusted input that may carry instructions, and assume some injections will succeed: limit what a hijacked model can do instead of trusting the prompt to resist. Where it comes from Direct: the user types "ignore your instructions"; mostly a nuisance unless the model holds secrets or privileges Indirect: instructions hidden in an email, web page, PDF or image the model processes ("forward this thread to…"); the dangerous one, because the victim is your user What limits the damage Least privilege: the summariser gets no tools; tools are scoped to the user's own data and the current task Confirmation: sending, paying, deleting and sharing need a confirmation screen built from the parsed arguments, not from model prose No secrets in prompts: assume the system prompt leaks; keys, other users' data and internal URLs never go in Output handling: render output as plain text; do not auto-load links or images it produces (data can leave in a URL query), and never pass it unchecked to WebView.evaluateJavascript, SQL or an Intent What helps but is not a defence fencing untrusted content and telling the model it is data, input classifiers, vendor safety settings: they lower the success rate, not to zero keep known injection samples in your eval set and re-run them on every prompt or model change

Parse only a complete object as final, and treat every earlier prefix as layout hints. Ask the vendor for schema-constrained output and stream the prefix into a mirror type with all-nullable fields. iOS 26 does it natively: a @Generable type streamed with streamResponse arrives as partial snapshots that fill in declaration order Parsing the prefix: truncate to the last completed member, close the open containers, decode leniently into the nullable mirror, and validate against the real type at the end Common mistake: taking a value from an unterminated token: a price that reads 12 because the stream stopped mid-1234 is wrong, not slow On screen: a skeleton shaped like the field, never a value that appears and is then corrected, and actions disabled until the object validates

WhileSubscribed(5_000) cancels the upstream 5 s after the screen stops collecting and runs it again for the next subscriber, so every return from the background starts a new paid generation from "", even when the summary had already finished. Why: collectAsStateWithLifecycle stops collecting below STARTED; after the timeout stateIn cancels the upstream, which here is the HTTP stream itself. A new subscriber collects the cold flow from scratch: a new request, and runningFold starts again from "", which blanks the screen Why the 5 s idiom fails here: it suits cheap, repeatable sources such as a Room query or location updates, where re-subscribing just re-reads current data; a generation is a paid side effect, and running it again costs twice and may give a different answer The fix: tie the generation to the ViewModel, not to the UI: SharingStarted.Lazily (or a launch in init writing into a MutableStateFlow) runs the stream once and keeps the result; viewModelScope still cancels it when the user leaves the screen, which is the stop you want Catch it next time: a runTest with a fake AssistantApi that counts calls: collect, cancel, advanceTimeBy(6_000), collect again, assert one call

Store the conversation as durable turns with ids and treat the ViewModel as a projection of them. On return, ask the server about the generation you already started instead of starting it again. What is durable: every turn is a row: id, role, text, status (pending, streaming, complete, failed); the screen renders from the store, not from memory What you replay: the last turn decides: a complete answer needs nothing, a user turn with no answer is re-sent with the same turn id, streaming or pending means asking the server about that generation id, an unknown one becomes failed with a retry Never twice: the user's turn: mint its id on the client and send it on every attempt, so a retry resumes instead of asking again; this matters most when the model can call tools that send or pay Never twice: a truncated answer as context: replayed as if finished, it makes the model continue it; mark it truncated or drop it from the window

Weights are content, not code: they arrive through the platform's asset-delivery channel, they version with the code that reads them, and they are a cache the system may reclaim, so the feature must keep working while they are missing. Your own weights: an on-demand asset pack through Play Asset Delivery (or an AI pack via Play for On-device AI, which can also target by device model) on Android, Background Assets or On-Demand Resources on iOS; all of them give you resume, integrity and a cache you did not write Common mistake: a plain GET of a gigabyte into Documents: no resume, no integrity check, it lands in the iCloud backup, and a half-arrived file becomes your crash While downloading: unmetered network by default, version the pack with its reader, and serve the feature from the server path meanwhile Deleting: the system purges these packs silently under disk pressure, so 'the model is gone' is a normal branch; keep your copies out of backup and delete them on sign-out

Split it: everything around the model is ordinary deterministic code tested against a fake model, and the model's quality is measured statistically with an eval set and graders, re-run on every prompt or model change. Around the model put the model behind an interface and test parsing, streaming, cancellation, fallback and the quota UI with a fake that replays recorded chunks, including malformed JSON and a mid-stream error pin the model version and sampling settings in config; an alias like latest changes behaviour under you The model itself Eval set: a few hundred inputs from real, consented and scrubbed traffic, plus known hard cases and injection attempts Graders: exact checks where possible (valid JSON, required fields, length, language), an LLM judge with a written rubric for quality, spot-checked by people Gate: run the evals in CI on every prompt or model change and compare with the baseline; a prompt is code and gets reviewed In production thumbs up/down and a report action, regenerate and cancel rates, time to first token and error rate per model version ship prompt and model changes behind remote config to a percentage of users, so a regression is reverted without a release log prompts only with consent and redaction