How do you choose between an on-device model (Gemini Nano, LiteRT) and a server model for a feature like "Summarise this thread"?

Default to the server path, because it works on every device, and use the on-device model only for bounded, short or private tasks, and only when the availability check passes.

This chat sends the whole history on every turn. What happens to cost and latency by turn 50, and how do you fix it?

Every turn re-sends and re-bills all earlier turns, so input tokens grow each turn, total spend grows roughly quadratically, time to first token creeps up, and at some point the request overflows the context window and fails.

What time to first token should a streaming AI screen aim for, and which ways of hiding the wait are honest?

Time to first token is the number that matters: about 100 ms feels instant, a second is fine if the tap was acknowledged, and after a few seconds people leave.

What do Google Play and the App Store require from an app with a generative AI feature?

Both treat generated output as your content: you must prevent and moderate harmful output and let users report it inside the app, and you must disclose and get consent before personal data goes to a third-party AI provider.

Your assistant summarises emails and can call tools. How do you defend it against prompt injection?

Treat everything the model reads as untrusted input that may carry instructions, and assume some injections will succeed: limit what a hijacked model can do instead of trusting the prompt to resist.

The app is killed mid-answer and the user comes back an hour later. What do you restore, what do you re-send, and what must never go twice?

Store the conversation as durable turns with ids and treat the ViewModel as a projection of them.

The on-device model weighs 1.2 GB and your app download cannot. How do those bytes get onto the phone, and when do you delete them?

Weights are content, not code: they arrive through the platform's asset-delivery channel, they version with the code that reads them, and they are a cache the system may reclaim, so the feature must keep working while they are missing.

The model answers differently on every run. How do you test an AI feature before release and know it still works after?

Split it: everything around the model is ordinary deterministic code tested against a fake model, and the model's quality is measured statistically with an eval set and graders, re-run on every prompt or model change.

Irbisa · Cheatsheet

AI Features in Mobile Apps

Q: This adapter turns an SSE token stream into a Flow. What is wrong with it?

It never calls `awaitClose`, so `callbackFlow` fails at runtime with `IllegalStateException` and nothing cancels the `EventSource`; it also never completes the flow on a normal end of stream and can silently drop tokens when the buffer fills.

Q: How do you run your own LiteRT (TensorFlow Lite) model on Android, and what does a GPU delegate change?

Load the `.tflite` file memory-mapped into one `Interpreter`, reuse it, and run it off the main thread.

On-device vs server inference, token streaming UI, structured output, latency, cost, privacy

Senior Developer · 17 questions

01
How do you choose between an on-device model (Gemini Nano, LiteRT) and a server model for a feature like "Summarise this thread"?
Default to the server path, because it works on every device, and use the on-device model only for bounded, short or private tasks, and only when the availability check passes.
```
interface ThreadSummarizer { suspend fun summarize(text: String): String } class OnDevice(context: Context) : ThreadSummarizer {  private val client = Summarization.getClient(    SummarizerOptions.builder(context)      .setInputType(InputType.ARTICLE)…
```
02
This chat sends the whole history on every turn. What happens to cost and latency by turn 50, and how do you fix it?
```
class ChatViewModel(private val api: ChatApi) : ViewModel() {  private val history = mutableListOf<Message>()   fun send(text: String) = viewModelScope.launch {    history += Message("user", text)    val reply = api.complete(history)   // the whole conversation, every turn    history += Message("assistant", reply)  }}
```
Every turn re-sends and re-bills all earlier turns, so input tokens grow each turn, total spend grows roughly quadratically, time to first token creeps up, and at some point the request overflows the context window and fails.

This adapter turns an SSE token stream into a `Flow`. What is wrong with it?

fun streamAnswer(client: OkHttpClient, request: Request): Flow<String> = callbackFlow {  val listener = object : EventSourceListener() {    override fun onEvent(eventSource: EventSource, id: String?, type: String?, data: String) {      trySend(data)    }    override fun onFailure(eventSource: EventSource, t: Throwable?, response: Response?) {      close(t)    }  }  EventSources.createFactory(client).newEventSource(request, listener)}

It never calls awaitClose, so callbackFlow fails at runtime with IllegalStateException and nothing cancels the EventSource; it also never completes the flow on a normal end of stream and can silently drop tokens when the buffer fills.

Find the bug: English answers stream fine, but Russian ones sometimes show `�` in the middle of a word.

// The endpoint streams the answer as plain text over chunked HTTP.fun streamAnswer(body: ResponseBody): Flow<String> = flow {  val buf = ByteArray(1024)  body.byteStream().use { input ->    while (true) {      val n = input.read(buf)      if (n == -1) break      emit(String(buf, 0, n, Charsets.UTF_8))    }  }}.flowOn(Dispatchers.IO)

Each network chunk is decoded on its own, so a Cyrillic letter or an emoji whose UTF-8 bytes straddle two chunks is cut in half, and both halves turn into �.

Your streaming answer stutters as tokens land, and the list keeps jumping while the text grows. What do you change?

One token is not one UI update.

// Deltas arrive far faster than a frame. Coalesce once, at the boundary.val answer: Flow<String> = flow {  val buf = StringBuilder()  api.streamAnswer(prompt).collect { delta -> buf.append(delta); emit(buf.toString()) }}  .flowOn(Dispatchers.Default)          // string building off the main thread…

The user hits back halfway through a generation. What actually has to happen on the client, and what does the server still need from you?

The cancellation must reach the socket: closing the response body is what stops the meter.

class AnswerViewModel(private val repo: AnswerRepo) : ViewModel() {  private var job: Job? = null   fun ask(turnId: String, prompt: String) {    job?.cancel()                      // a new question kills the old stream, and its cost    job = viewModelScope.launch {      // dies with the screen, not with the process…

07
What time to first token should a streaming AI screen aim for, and which ways of hiding the wait are honest?
Time to first token is the number that matters: about 100 ms feels instant, a second is fine if the tap was acknowledged, and after a few seconds people leave.
```
// TTFT is a different metric from total duration. Time the first chunk, not the last.suspend fun ask(prompt: String, sink: (String) -> Unit) {  val started = SystemClock.elapsedRealtime()  var firstAt: Long? = null  var tokens = 0  try {…
```

How do you run your own LiteRT (TensorFlow Lite) model on Android, and what does a GPU delegate change?

Load the .tflite file memory-mapped into one Interpreter, reuse it, and run it off the main thread.

class Classifier(context: Context) : Closeable {  private val compat = CompatibilityList()  private val gpu: GpuDelegate? =    if (compat.isDelegateSupportedOnThisDevice) GpuDelegate(compat.bestOptionsForThisDevice) else null   private val interpreter = Interpreter(…

09
What do Google Play and the App Store require from an app with a generative AI feature?
Both treat generated output as your content: you must prevent and moderate harmful output and let users report it inside the app, and you must disclose and get consent before personal data goes to a third-party AI provider.

How does function calling (tool use) work in a mobile assistant, and which tools should run on the device?

The model never executes anything: it returns a tool name with JSON arguments, your code validates and runs it, and you send the result back so the model can continue.

@Serializable data class CreateEvent(val title: String, val startEpochMs: Long, val minutes: Int) sealed interface ToolOutcome {  data class Result(val json: String) : ToolOutcome  data class NeedsConfirmation(val prompt: String, val run: suspend () -> String) : ToolOutcome}…

Why must LLM calls go through your backend instead of the app holding the vendor key, and what must that proxy own beyond hiding the key?

A vendor key inside the APK is public: anyone can extract it.

// WRONG: the client builds the prompt and carries the vendor key.// Anyone with the APK has the key; anyone with the key has your bill.val body = ChatBody(  model = "vendor-large",  messages = listOf(Message("system", SYSTEM_PROMPT), Message("user", userText)),)…

12
Your assistant summarises emails and can call tools. How do you defend it against prompt injection?
Treat everything the model reads as untrusted input that may carry instructions, and assume some injections will succeed: limit what a hijacked model can do instead of trusting the prompt to resist.

You ask the model for JSON and want fields to appear as they arrive. What do you do with a half-written object, and what does an unfinished field look like on screen?

Parse only a complete object as final, and treat every earlier prefix as layout hints.

@Serializable data class Recipe(val title: String, val minutes: Int, val steps: List<String>) // The streaming mirror: every field nullable, because every field may not exist yet.@Serializable data class RecipeDraft(  val title: String? = null, val minutes: Int? = null, val steps: List<String>? = null,)…

Find the bug: a user switches apps mid-summary, comes back, and the summary starts over on a blank screen; the bill shows two generations.

class SummaryViewModel(api: AssistantApi, threadId: String) : ViewModel() {  val summary: StateFlow<String> = api.stream(SummarizeRequest(threadId))    .runningFold("") { text, delta -> text + delta }    .stateIn(viewModelScope, SharingStarted.WhileSubscribed(5_000), "")} @Composablefun SummaryScreen(vm: SummaryViewModel) {  val summary by vm.summary.collectAsStateWithLifecycle()  MarkdownText(summary)}

WhileSubscribed(5_000) cancels the upstream 5 s after the screen stops collecting and runs it again for the next subscriber, so every return from the background starts a new paid generation from "", even when the summary had already finished.

15
The app is killed mid-answer and the user comes back an hour later. What do you restore, what do you re-send, and what must never go twice?
Store the conversation as durable turns with ids and treat the ViewModel as a projection of them.
```
@Entitydata class TurnEntity(  @PrimaryKey val id: String,        // minted on the client, reused by every retry  val conversationId: String,  val role: String,                  // "user" | "assistant"  val text: String,…
```
16
The on-device model weighs 1.2 GB and your app download cannot. How do those bytes get onto the phone, and when do you delete them?
Weights are content, not code: they arrive through the platform's asset-delivery channel, they version with the code that reads them, and they are a cache the system may reclaim, so the feature must keep working while they are missing.
```
// Play Asset Delivery: the model is an on-demand asset pack, not part of the APK.class ModelStore(private val packs: AssetPackManager) {  fun localPath(): String? = packs.getPackLocation(PACK)?.assetsPath()   // null = not here yet   fun ensure(activity: Activity, onReady: (String) -> Unit) {    localPath()?.let { return onReady(it) }…
```
17
The model answers differently on every run. How do you test an AI feature before release and know it still works after?
Split it: everything around the model is ordinary deterministic code tested against a fake model, and the model's quality is measured statistically with an eval set and graders, re-run on every prompt or model change.