In this post, we’ll go over what it means to sample data when debugging mobile apps, what the shortcomings are in that approach, and why a more comprehensive approach to diagnostic data collection helps mobile teams move faster.

Sampling (noun): Definition: the act, process, or technique of selecting a representative part of a population for the purpose of determining parameters or characteristics of the whole population.

Sampling is like a blood test; it’s used by mobile engineering teams to detect what’s wrong with with an app. However, also similar to a blood test, sampling doesn’t give the full picture. Despite that, mobile engineering teams haven’t improved upon this most basic of practices. Embrace has long maintained that the best practice should not be to sample the data, but to collect all of it so nothing escapes the mobile engineering team’s notice.

What happens during sampling?

When a mobile team is sampling, they set rules and collect incoming data when certain events trigger. This usually takes the form of logs and crashes. The same way a blood test can’t reveal everything about a body, sampling a subset of the whole data cannot possibly reveal everything wrong with the app. This means that when users have problematic sessions that don’t trigger the conditions for sampling, the mobile team is left in the dark.

Why has it been used so far?

If your mobile app is in its infancy, sampling does two things: it allows you to focus your resources and energy on the immediately actionable issues that can be caught by sampled data, and the upfront cost is much cheaper. By utilizing a sampling approach, the mobile team can determine what issues it cares about and write code that notifies them when those issues pop up.

This is what the mobile sampling process looks like:

  1. Determine what data the team cares about and wants to log: crashes, memory leaks, ANRs, etc.
  2. Collect all data related to these.
  3. A user reports something that is not being sampled.
  4. Add the new issue to your range of sampled data.
  5. Rinse and repeat over time, adding new issues to your sampling efforts.

But here’s why it’s bad in the long run:

  • A sample-only approach means that there are users whose sessions aren’t detected. Those users will have their issues stay undetected in the long run, and they will leave. Considering the CPI (Cost Per Install) of users is very expensive and LTV (Life-Time Value) of users requires them to stay on “in the long run” for conversion rates, any users not served by sampling are a flight-risk.
  • This is inherently reactive rather than proactive. Essentially, the sampling process is only updated when what the mobile team did not think to sample and capture is brought to their attention by users. This puts the mobile team in the unenviable position of fighting fires in production instead of discovering them earlier during development and testing.

What can we learn from sampling’s inadequacies?

The engineering team can better solve user issues the more data they collect (not to mention, the engineering team can solve issues faster the more data they collect).

This comes back to the current practice of sampling — collecting selective pieces of data and only updating that basket when new issues are discovered. It results in unhappy users experiencing problems that aren’t logged, and the engineering team only being aware of issues once they crop up in large enough percentages that a user actually submits a complaint.

What’s the answer then? The best practice should not be to sample; instead, companies that are using the sampling approach should simply capture all data. Everything happening in your app is directly correlated to the user experience, and that data can be used to make the app better. Instead of waiting until a user alerts the engineering team of an issue, the engineering team would already have the data when something goes wrong. This allows for proactive troubleshooting with full visibility into how drastic an issue is because the information of how many users it affects and to what degree it is affecting those users is immediately available to the engineering team for prioritization, and we know how important it is to align the engineering team to the correct KPIs.

If sampling, your KPIs may be skewed because the data collected is incomplete. If capturing all data, your KPIs will be better served because the engineering team has full visibility into the effects of each issue.

How Embrace helps mobile teams

Embrace is a data driven toolset to help mobile engineers build better experiences. We are a comprehensive solution that fully reproduces every user experience from every single session. Your team gets the data it needs to proactively identify, prioritize, and solve any issue that’s costing you users or revenue.

Request a demo and see how we help teams set and exceed the KPIs that matter for their business!