Skip to content

Why we measure on real devices

A four-year-old Android phone tells the truth. Your dev machine tells you what you want to hear.

Published

6 min read

A four-year-old budget Android phone tells the truth. Your dev machine tells you what you want to hear. The gap between those two voices is where most performance regressions live, undetected, until a customer sends a video of the app freezing.

This essay is about the device fleet we test on, why we picked each one, and what they have caught that emulators did not.

Why emulators lie#

The Android emulator on a recent Mac runs ARM code on Apple Silicon with hardware acceleration. It is fast. It uses the host's RAM. It uses the host's storage IO. It is, structurally, an order of magnitude faster than any real Android phone shipped this decade.

Performance work measured on the emulator passes. The same code, shipped to a Galaxy A series from 2021, freezes for three seconds on boot.

The same is true for iOS. The simulator runs native x86_64 / ARM code on the host, with the host's CPU and memory budget. It is not the device.

We have shipped fixes that "worked on simulator" five times that turned into bug reports within a week. We stopped trusting the simulator for performance. We use it for rendering and behaviour; performance numbers come from real hardware.

The fleet#

We keep four physical devices on a desk near the dev team. Each one represents a real customer segment.

DeviceYearWhy we have it
Galaxy A122020Budget Android, low RAM, slow eMMC. The "your app is slow" device.
iPhone 82017The oldest iOS we still support. Thermal throttles fast.
Pixel 6a2022Mid-tier modern Android. Most users approximate this.
Galaxy S232023High-end. Catches things that only happen at high refresh.

The first one is the most important. If a feature feels good on the A12, it will feel good on everything. The reverse is not true.

What real devices have caught#

Real bugs from real production:

A freeze on boot#

The list view's first paint involved a synchronous Drift query that read thousands of contacts to build the search index. On the M1 Mac, unnoticeable. On the A12, a visible freeze. We moved the index build to a post-paint isolate and the freeze went away.

A scroll jank that emulators couldn't see#

A list of invoices used Builder widgets that re-built more often than necessary. On any recent device, smooth. On the A12 with Emoji-rich text, the list dropped frames and the user felt it. Replacing Builder with cached widgets with == overrides got us back to smooth scrolling.

A battery drain during idle use#

The app was polling its sync endpoint every 5 seconds with no visibility check. On the iPhone 8 with old battery, this was visible in the system battery report within an hour. We moved to visibility-aware polling and the drain dropped sharply.

A connection-pool exhaustion under thermal throttling#

When the iPhone 8 hits thermal throttle, network sockets time out faster. Our retry logic wasn't accounting for this and was opening new sockets faster than they could close, exhausting the per-app limit. We saw it because the device got hot in a stress test.

How we run the tests#

Three layers.

1. Manual smoke on the slow device#

Before every release, the on-call engineer runs through a 15-minute smoke script on the A12 specifically: cold start, navigate to four key screens, do one create, one search, one offline-then-reconnect. Pass/fail by feel, not by number.

2. Automated metrics from the device fleet#

We run a nightly Maestro flow on each device that captures:

  • Cold-start to first-frame ms
  • Time to interactive on the list view
  • Memory at idle after 5 minutes
  • Memory after 50 navigations through the app
  • Battery delta during a 10-minute scripted session

Numbers go to a dashboard. Regressions vs. last week's median trip an alert.

3. Real-user telemetry#

In production, we sample timing on key actions and ship them to a metrics store, tagged with device class. We can compare "list-view first-paint p95" between Galaxy A series and iPhone 13. The fleet predicts what we will see in RUM; RUM is the source of truth.

What "device class" means in our telemetry#

We do not record specific device models in telemetry (PII concern). Instead, every event carries a coarse class:

type DeviceClass =
  | 'budget-android'   // <4 GB RAM or known slow eMMC
  | 'mid-android'      // 4-8 GB RAM
  | 'flagship-android' // >8 GB RAM
  | 'old-ios'          // <iPhone 11
  | 'mid-ios'          // iPhone 11-13
  | 'flagship-ios';    // >iPhone 13

Six classes cover the world for our purposes. We stratify metrics by class because aggregating across them hides the picture. p95 on "flagship-android" is different from p95 on "budget-android" by an order of magnitude.

What this discipline costs#

Time, mostly. The nightly fleet run takes about 40 minutes (parallel across devices) and surfaces noise that has to be triaged. The manual smoke before each release is 15 minutes per release. Devices need replacing as their batteries degrade and as the OS updates beyond what the app supports.

The cost is low compared to the cost of shipping a regression that makes the app feel broken on the budget Android segment, which is where most of our growth is.

The bigger principle#

Performance is what the user feels on the device they have. If your measurement happens on a device they don't have, you are measuring something else. Most performance dashboards are dishonest because they aggregate over the dev team's hardware and miss the segment that matters most.

Buy the slow phone. Keep it on your desk. Run the app on it before shipping. Most of what looks like a "performance optimisation opportunity" is actually a budget mismatch between the device the code was written on and the device it will be used on.

The next post is on the forgotten cost of connection setup. Or browse the writing index for other directions.

Read this article in Arabic · العربية

Share this article

Ali Haitham Yosef

Ali Haitham Yosef, a software developer and company owner in Damascus. Working since 2015, with more than 600 projects delivered.

About Ali

Does your project look like what you read?

Write what you need in your own words, and we decide together whether what I offer fits your company.

Start a conversation