Performance
Why we measure on real devices
A four-year-old Android phone tells the truth. Your dev machine tells you what you want to hear.
Loading essay…
A four-year-old Android phone tells the truth. Your dev machine tells you what you want to hear.
Loading essay…
A four-year-old budget Android phone tells the truth. Your dev machine tells you what you want to hear. The gap between those two voices is where most performance regressions live, undetected, until a customer sends a video of the app freezing.
This essay is about the device fleet we test on, why we picked each one, and what they have caught that emulators did not.
The Android emulator on a recent Mac runs ARM code on Apple Silicon with hardware acceleration. It is fast. It uses the host's RAM. It uses the host's storage IO. It is, structurally, an order of magnitude faster than any real Android phone shipped this decade.
Performance work measured on the emulator passes. The same code, shipped to a Galaxy A series from 2021, freezes for three seconds on boot.
The same is true for iOS. The simulator runs native x86_64 / ARM code on the host, with the host's CPU and memory budget. It is not the device.
We have shipped fixes that "worked on simulator" five times that turned into bug reports within a week. We stopped trusting the simulator for performance. We use it for rendering and behaviour; performance numbers come from real hardware.
We keep four physical devices on a desk near the dev team. Each one represents a real customer segment.
| Device | Year | Why we have it |
|---|---|---|
| Galaxy A12 | 2020 | Budget Android, low RAM, slow eMMC. The "your app is slow" device. |
| iPhone 8 | 2017 | The oldest iOS we still support. Thermal throttles fast. |
| Pixel 6a | 2022 | Mid-tier modern Android. Most users approximate this. |
| Galaxy S23 | 2023 | High-end. Catches things that only happen at high refresh. |
The first one is the most important. If a feature feels good on the A12, it will feel good on everything. The reverse is not true.
Real bugs from real production:
The list view's first paint involved a synchronous Drift query that read 8,000 contacts to build the search index. On the M1 Mac, 12 ms. On the A12, 380 ms. We moved the index build to a post-paint isolate and the freeze went away.
A list of invoices used Builder widgets that re-built more often
than necessary. On any recent device, 60 fps. On the A12 with
Emoji-rich text, the list dropped to 35 fps and the user felt it.
Replacing Builder with cached widgets with == overrides got us
back to 58 fps.
The app was polling its sync endpoint every 5 seconds with no visibility check. On the iPhone 8 with old battery, this was visible in the system battery report within an hour. We moved to visibility-aware polling and the drain dropped to 4 %.
When the iPhone 8 hits thermal throttle, network sockets time out faster. Our retry logic wasn't accounting for this and was opening new sockets faster than they could close, exhausting the per-app limit. We saw it because the device got hot in a stress test.
Three layers.
Before every release, the on-call engineer runs through a 15-minute smoke script on the A12 specifically: cold start, navigate to four key screens, do one create, one search, one offline-then-reconnect. Pass/fail by feel, not by number.
We run a nightly Maestro flow on each device that captures:
Numbers go to a dashboard. Regressions vs. last week's median trip an alert.
In production, we sample timing on key actions and ship them to a metrics store, tagged with device class. We can compare "list-view first-paint p95" between Galaxy A series and iPhone 13. The fleet predicts what we will see in RUM; RUM is the source of truth.
We do not record specific device models in telemetry (PII concern). Instead, every event carries a coarse class:
type DeviceClass =
| 'budget-android' // <4 GB RAM or known slow eMMC
| 'mid-android' // 4-8 GB RAM
| 'flagship-android' // >8 GB RAM
| 'old-ios' // <iPhone 11
| 'mid-ios' // iPhone 11-13
| 'flagship-ios'; // >iPhone 13Six classes cover the world for our purposes. We stratify metrics by class because aggregating across them hides the picture. p95 on "flagship-android" is different from p95 on "budget-android" by an order of magnitude.
Time, mostly. The nightly fleet run takes about 40 minutes (parallel across devices) and surfaces noise that has to be triaged. The manual smoke before each release is 15 minutes per release. Devices need replacing as their batteries degrade and as the OS updates beyond what the app supports.
The cost is low compared to the cost of shipping a regression that makes the app feel broken on the budget Android segment, which is where most of our growth is.
Performance is what the user feels on the device they have. If your measurement happens on a device they don't have, you are measuring something else. Most performance dashboards are dishonest because they aggregate over the dev team's hardware and miss the segment that matters most.
Buy the slow phone. Keep it on your desk. Run the app on it before shipping. Most of what looks like a "performance optimisation opportunity" is actually a budget mismatch between the device the code was written on and the device it will be used on.
The next post is on the forgotten cost of connection setup. Or browse the writing index for other directions.