You shouldn’t expect users to deliver detailed bug reports

“Would you like to send a bug report to the developer to help them improve the game?”

Ahh, report requests — what better way for the developer to admit they have no idea why the user just suffered a crash or bug than to ask the user directly about what they were doing before the issue?

Mobile apps can experience many different bugs and crashes, but this experience can be far worse for mobile games with users that are looking for an interactive experience. To fix this issue, the mobile engineering team has two primary methods to gather information on how to improve the app: their own tooling or user-submitted bug reports.

In this post, we’ll discuss why relying heavily on bug reports for debugging purposes is unrealistic, unpredictable, and inefficient. Let’s first run through the possible scenarios when it comes to user-submitted bug reports.

Best-case scenario

Every user fills out comprehensive bug reports and documents exactly what went wrong. They do the heavy lifting of issue discovery for the engineering team.

But what does this entail on the user’s part? After all, the work is being offloaded onto the users, right?

To accomplish that, the user-submitted bug reports must contain the following:

• What was the bug?
• Severity and priority
• Description of what happened
• Environment for the app (OS, hardware, firmware, etc.)
• How to reproduce the bug (detailed step-by-step instructions)
• What happened vs. what was expected
• Attachments, if any (logs, stack trace, screenshots, videos)
• Contact details for the user if the engineering team has follow-up questions

For many mobile game companies, this best-case scenario is a dream come true. If you have users that will commit to submitting these sorts of bug reports you have produced a game they absolutely love and are invested in its continued improvement and success. Thus, your users significantly reduce the engineering resources required for debugging…right?

But this best-case scenario has a hidden cost: screening. If your mobile app is in the position of receiving bug reports from every single user, then the engineering team will need a method to process it all.

For example, think about how large tech companies test their software. Perhaps they make all their employees test early releases on their personal phones as part of their job, simulating a small amount of user-submitted bug reports. Because this is part of their job, these employee-users are submitting the ideal bug reports in order to improve the app as much as possible.

But the amount of users drastically outnumbers the engineering team, and if each user submits a bug report then the engineering team needs a way to screen, filter, and prioritize these reports or risk being overwhelmed. Yes, engineering gets perfect bug reports, but then more resources are required just to screen them!

This means that the best-case scenario is a nightmare for small companies because they cannot afford to do this. Without the tooling to effectively categorize issues, you’ll need engineering time dedicated to triaging and assigning bug reports manually.

Additionally, it is not a reliable recipe for success. Many users would rather uninstall a game than bother filling out a bug report.

Over-reliance on user-submitted bug reports is signaling to users that you’re not being proactive about issue detection and resolution. Expecting users to do unpaid QA work is a recipe for resentment if heavily relied upon and burns the goodwill of your users.

Some mobile games even have a dedicated forum space for user-submitted bug reports. But this doubles as a hall of shame for issues the engineering team should have proactively caught and resolved, sometimes resulting in lawsuits.

Let’s be honest: No one wants to play a game that they’re not sure is being properly maintained.

Middle-case scenario

A trickle of users submit bug reports that are usable. They lack important data, but they are robust enough to point engineers in the right direction. This is a more reasonable scenario a mobile team can hope for from invested users. These bug reports are characterized by:

• Incomplete information
• Automated logs
• Occasional misleading information due to user error

This all assumes that the engineering team built a bug and crash reporting system for the users with the least amount of friction, and yet it is still undesirable for both sides.

If the users fill out a crash report with incomplete information, the best the engineering team can hope for is a promising lead.

If the developers only receive automated logs with information they had the foresight to capture, then the developers are missing out on any user input that could provide additional context. In many cases, this user context is critical to swiftly resolving a problem.

Worse, if users incorrectly fill out crash reports with misleading information, engineering time can be wasted chasing false leads.

This is before you recognize that not all users will fill out bug reports. Let’s be generous and assume that half of your users are willing to fill out bug reports. If bugs are evenly distributed across your users, that means your engineering team has severe blind spots into what issues are plaguing the app.

The worst issues — the ones that prevent a user from opening the app at all or logging in — also prevent the user from reporting the bug in your nice, well-built bug report UI. So by definition you are blind to the worst user problems that can plague your app if you are overly reliant on user-submitted bug reports.

With incomplete data, your engineering team is neither proactive nor efficient in issue remediation, and this has long-term ramifications for churn as well as the desirability of your game.

Worst-case scenario

Users don’t report anything and just uninstall your app.

Technical churn is the most realistic result of relying on user-submitted bug reports to gather feedback on your mobile app. Because these users encountered a game-breaking bug or experience-ruining crash, they determined your app was no longer of any value to them and won’t give it a second chance.

That unreported bug or crash will be hidden from the engineering team’s radar and will haunt the app, staying long enough for another user to encounter it. And if they also choose to uninstall instead of reporting the issue, this sequence of events will repeat itself ad nauseam causing your mobile app to bleed users to technical churn until it is resolved.

This happens far more than expected and it is also most likely to be untraceable. Even if you send a survey asking every user why they uninstalled your app, what percentage will respond? In their response, do you think they will give the engineering team anything of value besides a vague “the game was buggy” statement? No — over 100 new games are released every week on the App Store, and from the user’s perspective their precious time and attention can be better spent on installing a new game.

Moreover, even if users answered it was an issue that caused them to uninstall but the issue is undetectable by your monitoring solution, the engineering team is only left with the knowledge that something is broken. Not how to fix it.

This is because many problems elude traditional error monitoring platforms:

• Slow startups
• App Not Responding (ANRs)
• Memory leaks
• Abandoned key user flows
• Input lag
• Network lag

In search of better data

All the above scenarios suffer from the same underlying problem: your mobile engineers are put in the position of being reactive instead of proactive. They must discover issues from user complaints instead of from their tooling, and this is not a recipe for retention in the gaming world. After all, the mobile game company might not know a bug is plaguing the app until enough users report it, shown in this Pokemon Go fan wiki page that details every single bug.

So even the hypothetical best–case scenario is a worst-case scenario for companies.

All of this is to say: the gathering of data and feedback for the engineering team cannot and should not ever depend on user-submitted reports.

But what is the solution? Tooling built to collect comprehensive, actionable data for mobile engineers to understand exactly how users are experiencing the app.

Instead of relying on the charity of users to provide visibility, you should get it out of the box. That way, your mobile engineers can identify, prioritize, and solve issues faster before they affect your users.

How Embrace helps mobile gaming teams

Embrace is a data driven toolset to help mobile engineers build better experiences. We are a comprehensive solution that fully reproduces every user experience from every single session. Your team gets the data it needs to proactively identify, prioritize, and solve any issue that’s costing you users or revenue.