Mobile CI/CD (Cross-platform)

CI builds and tests every merged change; continuous delivery keeps every green build releasable behind one manual step; continuous deployment ships every green build to users automatically. Mobile stops at delivery because store review and user-controlled updates sit in between. CI: every PR and every merge runs lint, unit tests and a build, so main is never broken for long Delivery: the pipeline signs the build and puts it on a test track (Play internal testing, TestFlight); a person decides when it reaches production Why not deployment: review takes hours to days, users update when they choose, and a shipped binary cannot be recalled, so every release lives on devices for months What mobile does instead: automatic delivery to testers on every main build, plus fast server-side changes through feature flags and remote config

A mobile pipeline is really two: a fast lane that must finish while the reviewer is still looking at the PR, and a slow lane that may take an hour. Fast lane, every PR: lint and static analysis, then unit tests; no emulator, no signing, no uploads Slow lane, on main or nightly: UI tests on an emulator, a signed release build, upload to TestFlight or Play internal testing, symbols, staged rollout Keep the PR build under fifteen minutes: past that people stop reading the results Cache, and stay on Linux: an uncached build is mostly downloading, and a macOS minute costs several times more Signing material never lives in the repo: it comes from encrypted secrets and never reaches a log

Fastlane is a Ruby toolkit that turns the release ritual (build, sign, upload symbols, ship to a test track) into named lanes you run identically on a laptop and in CI. Build and sign: build_app on iOS, the gradle action on Android **match: synchronises iOS certificates and provisioning profiles through a private encrypted git repository, so every machine and the runner share one set Store auth: an App Store Connect API key and a Play service account, never a personal login (see the pipeline-authentication question) Distribution**: upload_to_testflight, upload_to_play_store, firebase_app_distribution, staged rollout percentages included Symbols: upload_symbols_to_crashlytics, so release crashes are readable

The runner is almost always right: something the build needs is not in the repository, or has a different name once the filesystem stops ignoring case. Untracked files the build reads: google-services.json, an xcconfig with keys, a .env; the pipeline must write them from secrets and fail loudly when they are missing An over-broad ignore rule: *.jar swallows the Gradle wrapper; git check-ignore -v names the guilty line Case: macOS ignores it, the Linux runner does not, so a mis-capitalised import compiles at home and fails in CI Lockfiles and the executable bit: commit Podfile.lock and Gemfile.lock, and set +x on gradlew through git Reproduce it in thirty seconds: clone into a temp directory and build there

You sign the AAB with your own upload key; with Play App Signing, Google keeps the app signing key and re-signs the APKs it delivers, so the key that devices verify never sits on a laptop or a CI runner. Two keys: the upload key proves the upload came from you; the app signing key is the app's identity on every device, and every update must carry it Why it matters: a lost or leaked upload key can be reset in the Play Console; a lost app signing key without Play App Signing means the app can never be updated again Required for AAB: new apps have had to ship AABs since August 2021, and an AAB only works with Play App Signing, because Google builds and signs the per-device APKs On CI: the upload keystore is a base64 secret, decoded at build time and deleted afterwards; passwords come from environment variables, never from a committed gradle.properties APK signature schemes: for a directly distributed APK, apksigner writes v1 to v4, zipalign must run before signing, and any later edit breaks the signature; an AAB is signed with the upload key only

Each store gets a machine credential owned by the team rather than a person: a Google Cloud service account invited into the Play Console, and an App Store Connect team API key. Both live in the CI secret store and nowhere else. Play: create a service account, enable the Google Play Android Developer API, invite the account's email under Users and permissions, and grant only what the pipeline does: release to testing tracks, and production only in the job behind the protected environment, never financial data App Store Connect: a team API key (the .p8 file plus key id and issuer id) with the narrowest role that can upload builds and manage TestFlight; the private key can be downloaded exactly once Why not a person's login: an Apple ID two-factor prompt cannot be answered on a runner, and the pipeline breaks the day that person leaves Storage: the service-account JSON and the .p8 go into a protected environment's secrets; fastlane reads them through json_key_data and key_content, so no file is written to disk Rotation: both are long-lived credentials with no second factor; rotate them on a schedule and whenever someone with access leaves

Play re-signs your upload with the app signing key, so the certificate fingerprint on a user's device is Google's and not your upload key's, and every integration keyed to that fingerprint has to know both. Register both keys' fingerprints: the OAuth client and restricted API keys want the SHA-1, assetlinks.json wants the SHA-256, and Firebase takes both; copy them from the Play Console app signing key certificate and add them beside your upload and debug ones App Links fail silently: a wrong fingerprint in assetlinks.json gives no error, the link just opens in a browser The installed app is not your APK either: Play splits the AAB by ABI, density and language, so sideloading only the base APK crashes at startup Test what users actually get: internal app sharing serves the real Play-signed split set from a link, with no version code rules and no review wait Locally: bundletool build-apks --connected-device builds exactly the splits that device would receive

The runner fetches the team's certificates and provisioning profiles read-only with fastlane match and imports them into a temporary unlocked keychain; signing then works only if the certificate, profile, entitlements and bundle id all agree. Getting it onto CI: fastlane match keeps certificates and profiles in a private encrypted repository, and the runner installs them read-only, so it can never regenerate and thereby revoke the team's certificates Using it headless: import into a temporary keychain that is unlocked, has no relock timeout and has its key partition list set; skip the partition list and codesign fails with "User interaction is not allowed". setup_ci creates and unlocks the temporary keychain; match sets the partition list when it imports Authenticate with an App Store Connect API key: two-factor makes an Apple ID password unusable unattended Entitlements must match the profile exactly: a regenerated profile can silently drop push or App Groups, and the mismatch surfaces at upload, after the slow part **Keep -allowProvisioningUpdates off CI: it lets the build machine change the team's profiles; signing state should live and die with the job An expired certificate breaks new builds, not installed App Store apps**: which is why it always turns up on a deadline

Android uses productFlavors, iOS uses xcconfig-backed build configurations selected by schemes, and Flutter's --flavor drives both; each added flavor multiplies what CI builds, signs and tests. Android: productFlavors times buildTypes gives variants such as stagingDebug; applicationIdSuffix lets them install side by side, and src/<flavor>/ overrides resources and google-services.json iOS: build configurations backed by xcconfig files, selected by schemes; a different bundle id per environment also means its own provisioning profile Flutter: --flavor maps onto both native mechanisms, and --dart-define-from-file passes the per-environment constants to Dart CI cost: the PR lane builds one variant (say devDebug), the release lane builds prodRelease; building every variant on every PR is how a pipeline doubles without anyone deciding it should Common mistake: flavors for things that should be runtime flags; a flavor is for a different app identity, not for a feature toggle

The build number is pipeline state, not source a human edits: CI derives a value that only ever goes up. Where it comes from: ask the store for the last one and add one (latest_testflight_build_number, google_play_track_version_codes), or use a counter that cannot go backwards, such as the CI run number A number is burned forever: deleting a TestFlight build or a Play draft does not free the value Common mistake: two people bumping by hand on two branches; the collision only shows at upload, after the slow part

Almost every unexplained CI failure comes from something nobody pinned: a runner image, a toolchain, or a dependency that resolved to a version which did not exist last week. Runner images move: a latest label changes the default Xcode, or the JDK and NDK; pin the label and pick the toolchain explicitly inside it Xcode is not just a compiler: it sets the Swift version, the simulator runtimes and what the store accepts on upload The JVM side is a matrix: Java, Gradle, AGP and Kotlin; commit the wrapper and declare a Java toolchain so the runner's default JDK is not deciding Ruby and fastlane: commit the lockfile and run through bundle exec; a bare fastlane resolves to whatever the image ships Floating versions: a dynamic Gradle version or a Swift package tracking a branch turns somebody else's release into your outage Stay ahead: a bot raising upgrades as PRs, and a nightly canary on latest that fails on nobody's deadline

Upload the dSYMs, the R8 mapping file and native debug symbols for that exact build from the job that produced the binary, before it ships; a rebuild does not recreate them. What to keep: dSYM bundles on iOS, R8's mapping file plus native debug symbols on Android, and the split-debug-info output for an obfuscated Flutter build Upload in the job that produced the binary: compiler output is not identical across machines, so the mapping for build 4213 exists exactly once iOS: since Xcode 14 dropped bitcode, App Store Connect no longer recompiles your binary, so the dSYMs from your own archive are the ones to upload; download_dsyms only matters for old bitcode builds Android: the mapping has to go both to the Play upload and to your crash vendor; they do not share the file Common mistake: a version or build number that does not match the symbol upload: the file is there, the stack is still hex, and nobody sees why Keep old versions' symbols: old versions crash too, and those are the crashes you are trying to fix

Make a red build reliably mean a broken app: remove the cause of each flake, retry a single test rather than the job, and quarantine repeat offenders with an owner. Where they run: Android emulators on accelerated Linux runners are cheapest, and Gradle Managed Devices make the device a declared build input; iOS simulators come with the macOS runner Real devices nightly, not per PR: they earn their cost on manufacturer skins, biometrics and low-memory hardware Remove the cause before you add a retry: the test orchestrator with data cleared between tests, animations off, the network stubbed, the suite sharded across runners Retry one test, never the whole job: a job retry hides the flake, doubles the bill and teaches everyone to press rerun Quarantine with an owner and a ticket: and track flake rate per test; a suite whose red builds are routinely ignored is worse than no suite Keep the evidence: video, device log and failure screenshots as artifacts

A rebuild is a new binary (a different runner image, a dependency that resolved differently, a new build number), so what ships is not what anyone tested. Compile once, in one job, and publish the .ipa or .aab, the symbols and the resolved versions as one immutable artifact named by its build number. Later stages move that artifact; nothing recompiles. iOS: the TestFlight build is the App Store candidate; submit that build number for review instead of uploading a fresh binary Android: promote the same AAB between tracks by version code; the Play API takes a track promotion with no artifact attached Configuration is compiled in: a staging binary cannot be promoted to production, so the candidate is built with production config Common mistake: a deploy job that runs build again; it hides because both builds usually work Keep artifacts longer than the release cycle: the usual 90-day default leaves a hotfix on an old release nothing to promote

Trunk-based development with unfinished work behind flags, a release branch cut on the train's schedule that receives only cherry-picked fixes, and store builds made from tags. Trunk-based: short branches into main, main always releasable, unfinished work behind a flag; a three-week feature branch is a merge conflict with a deadline attached Cut a release branch on the train's schedule: after that only cherry-picked fixes land on it, main keeps moving, and every hotfix is merged back or you ship the same bug next train Build the store artifact from a tag: that tag is what a hotfix branches from three weeks later A feature flag is the rollback: you cannot recall an installed build

Cancel superseded runs, skip jobs the change does not touch, build debug on PRs, cache by lockfile, shard the tests and move everything that does not need Xcode to Linux; a build scan or Xcode's build timing summary says which pays first. Stop work nobody asked for: a concurrency group per branch with cancel-in-progress kills runs a new push already superseded, and path filters keep a README change from building two apps Debug builds on PRs, release builds on main: coverage, screenshots and the full device matrix belong in the nightly lane Cache deliberately: key on the lockfile hash with a fallback prefix, cache the Gradle home, SwiftPM and CocoaPods, and turn on Gradle's configuration and build caches; do not cache DerivedData wholesale Shard instead of lengthening: ten minutes of tests on six runners is a three-to-four-minute job once setup is counted Spend where it works: anything that does not need Xcode goes on a Linux runner, and Gradle Managed Devices with ATD images make emulator boot faster and identical on every run Watch the trend: publish job duration; a pipeline that rots ten seconds a week is how it reached forty minutes

Secrets have to be unreachable from any code a stranger can change: fork pull requests run without them, and release jobs sit behind a human approval. **pull_request_target is the hole: it runs in the base branch's context with secrets and write scope, so checking out the pull request's head and building it hands an attacker everything; check out the base only, or split untrusted build from trusted publish A build script is code: a pull request that edits a Gradle file, the Gemfile, the Podfile or a fastlane action runs on your runner before anyone reads the diff Pin third-party actions to a commit SHA: a tag can be moved under you, which is exactly how several supply-chain incidents worked Least privilege**: explicit permissions on the workflow token, real secrets in a protected environment with required reviewers, and fastlane match read-only Ephemeral runners only: a persistent self-hosted runner keeps whatever the last job left behind, including a poisoned cache Log masking is not a control: base64 defeats it, so treat any secret that reached a log as burned

On iOS you can revoke and reissue everything, because Apple re-signs what users install from the App Store. On Android, if Play App Signing holds the app signing key, only the upload key leaked and you reset it; if not, the key is the app's identity and cannot be fully replaced. iOS: revoke and reissue: App Store apps were re-signed by Apple, so installed copies are unaffected; only your own builds break until profiles are regenerated, and fastlane match nuke distribution revokes them while the next non-readonly match appstore run issues new ones The iOS exception: ad hoc and in-house enterprise builds are validated against your certificate, so revoking stops them installing Rotate the App Store Connect API key too: a .p8 is a complete credential with no password behind it Android with Play App Signing: the leaked key is only your upload key; request an upload key reset and users notice nothing, because Google's key never left Google Android without it: the leaked key is the app's identity: anyone can sign an update that installed copies accept when sideloaded. Enrolling in Play App Signing does not retire it, because you hand the same key to Google. The remedy is a key upgrade through Play (new key for new installs, v3 proof-of-rotation for existing installs on Android 9+); the old key stays hostile for older devices Any Android key change: re-register every fingerprint-derived integration: OAuth clients, restricted API keys, App Links

Separate shipping from launching: get both binaries to users ahead of the date with the feature dark, then turn it on from the server once both are actually out. Store dates cannot be the mechanism: review latency is not yours to control, and even after approval a phased or staged rollout means not everyone holds the new binary on Tuesday Hold approved builds: App Store Connect's manual release and Play's managed publishing both park an approved release until you publish it Gate the feature on a remote flag: default off, shipped in both binaries a release or two ahead; that flag is also your rollback, which matters because a build cannot be recalled Keep the backend serving the previous version: the flag decides behaviour, and the API cannot assume a client version Common mistake: a flag with no safe default; cache the last known value and fall back to off, or a config outage becomes a feature outage Expedited review is an emergency lever: spend it on a marketing date and you will not have it during an incident