How often does your app crash?

It’s a great question because there are several nuances to it. For one, the question is better phrased as “How often do your users have a poor experience?” This is because true “crashes” (those cases where the app suddenly closes and there is a stack trace for the closure) represent only a small fraction of cases that the user would label as a crash.

We’ve found that OOMs (Out of Memory), which are instances where the app suddenly closes and doesn’t leave a stack trace, affect users 5 to 10 times more frequently as crashes and frustrate the user just as much as a technical “crash.”

Furthermore, the question is even more complicated by the fact that user terminations (situations where the user force closes an app) aren’t crashes, but in many cases they might as well be. If the user is stuck on a purchase page with a never ending spinner and force closes the app, they didn’t experience a crash. But, is what they experienced the outcome of having to exit the app and the consequent feeling of frustration, any better?

User terminations are by far the most frequent type of non-normal user session. We’ve found that on iOS, on average, true crashes represent < 0.1% of sessions, OOMs are around 0.25% of sessions, and user terminations are approximately 4% of sessions. Many of those user terminations are “benevolent” user terminations, cases where the user simply wants to clear all of the apps running in the background on their phone.

How do you find “malevolent” terminations?

A starting point to see if there is a problem is to compare recent days of activity or a recent app version to the typical app baseline. Historically, if 4% of sessions end in a user termination for an app, but the most recent version has 8% of sessions ending in user termination, then you can infer that the design of the app has changed in such a way that user terminations are more frequent. Note here that the differential is more important than the magnitude. We monitor several apps that have a baseline rate of 10% user terminations because users suspect them of draining battery in the background. The 10% baseline rate is a less important fact for those apps than when they see a spike in those terminations to 15%.

Once you know your app has too many user terminations, how do you find what’s causing them?

By focusing on the last view that a user saw before closing the app, and comparing the distribution of last views for user termination sessions with last views overall, you can quickly see problem areas in the app. For example, consider two eCommerce apps with just 3 screens, App A (a healthy app) and App B (an app with a broken purchase screen):

App A (95% User Terminated Free)

Last View Correlation Difference User Terminated % Overall %
Item List Screen Low + 1% 76% 75%
Add to Cart Screen Low – 1% 19% 20%
Purchase Screen Low 0% 5% 5%

App B (92% User Terminated Free)

Last View Correlation Difference User Terminated % Overall %
Item List Screen Low – 25% 50% 75%
Add to Cart Screen Low + 5% 25% 20%
Purchase Screen High +20% 25% 5%

App B has both a higher user terminated rate and a disproportionate number of those user terminations occurring on the purchase screen. It would make sense for developers of App B to investigate the purchase screen first. Furthermore, it would be helpful for the developers to view the user timelines of any of those terminated sessions, allowing them to see if there are additional network calls or user actions that provide further hints into what occurred.

Remember to start thinking outside of blanket “crashes” and focus more on the possible reasons that users are having a poor experience with your app. Investigate the multitude of different errors and failures that can occur to identify the real issues. Once you have a clear map of the problems, you can start solving, which will result in better experiences for users, potentially leading to action completions, longer sessions, and greater engagement and retention in the long-run.