How does an ANR work?

Application Not Responding (ANR) errors are some of the most frustrating to encounter and often difficult to debug. Worse yet, if you can’t keep your ANR rate under control, your app will be downranked and less discoverable in the Google Play Store.

While Android documentation provides guidance on ANRs, we’ve found that it only scratches the surface on exactly how these particular errors work in the wild.

To help you get a better handle on how to detect and capture ANR data, we asked our Android architect Jamie Lynch to describe in detail how the Android OS monitors, processes, and triggers ANRs.

An ANR begins when the Android framework calls AnrHelper.appNotResponding().

This is called from four places – once for each of the four fundamental components of Android.

Fundamentally, the Android Framework triggers an ANR when a component has not responded to the system within an expected time threshold. The framework usually implements this by having a background thread schedule a delayed call to AnrHelper.appNotResponding(). If work is finished before the threshold, this call is cancelled; if work is not finished, an ANR is born.

A BroadcastReceiver triggers an ANR if it doesn’t complete onReceive within 10 seconds. This is achieved by posting then removing a Runnable to the Handler of ActivityManagerService.

BroadcastReceiver components have a couple of unique exemptions from ANRs:

1. If the system is booting, as some early broadcasts setup system services.
2. If the timeoutExempt flag is set on the BroadcastRecord. This only seems to be enabled for the Intent.ACTION_PRE_BOOT_COMPLETED broadcast.

ANRs can trigger in two places for a Service:

1. A foreground service triggers an ANR if the component doesn’t call startForeground in under 10s. Interestingly, this is configurable via the Build.HW_TIMEOUT_MULTIPLIER property (although manufacturers aren’t supposed to alter this). Another interesting point is that doing this will crash the app, and that the main thread doesn’t necessarily have to be blocked for this to happen.
2. A regular service triggers an ANR if the component doesn’t start/bind in under 20s, or in under 200 seconds. The threshold is lower if the app is in the foreground.

In both cases the ANR is triggered by posting then removing a Runnable to the Handler of ActivityManagerService.

A ContentProvider triggers an ANR if getProviderMimeType() doesn’t respond within 1000ms. This is achieved by posting then removing a Runnable to the Handler of ActivityManagerService.

Developers can avoid ContentProvider ANRs by using the non-deprecated getProviderMimeTypeAsync method.

An Activity triggers an ANR if input dispatching times out. This is the most complicated of all the components:

• The WindowManagerService creates an InputManagerCallback.
• The native InputDispatcher polls to check whether an ANR has occurred:
  • The native AnrTracker returns the time when the first event on the connection queue would trigger an ANR, or LONG_LONG_MAX if the queue is empty.
  • The InputDispatcher checks if the current time exceeds the returned time. If this is the case it calls into InputManagerService.
• The InputManagerService forwards the call onto InputManagerCallback.
• InputManagerCallback forwards the calls onto AnrController.
• AnrController maps an app/input channel token onto a ProcessRecord or PID. This allows for the misbehaving process to be identified, and the AnrController then calls inputDispatchingTimedOut in either AnrRecord or ActivityManagerService.
• AnrRecord.inputDispatchingTimedOut() calls ActivityManagerService.inputDispatchingTimedOut().

1. The developer can trigger an ANR directly via the ActivityManagerService by calling appNotResponding(). This API was added in Android 11 for testing purposes.
2. Triggering an ANR causes all input events to be dropped (until the ANR resolves).

If the NDK layer blocks the main thread this would trigger an ANR if input dispatching times out.

SIGQUIT does not appear to be raised anywhere obvious from within the native code so the input dispatch scenario is the only way an ANR can occur.

It is very important to note that BroadcastReceiver and Service components always run on the main thread. ContentProvider runs on the calling thread (which is usually the main thread).

In a production app there might be dozens of these components performing work sequentially on the main thread. This can easily starve the main thread so that an Activity has no time to respond to input events, leading to ANRs that originate from within an Activity but are actually a problem in a different component!

For example, if a BroadcastReceiver blocks for 4.8 seconds and an Activity then blocks for 0.3 seconds, the Activity will be the culprit in the ANR stacktrace. This can make ANR error reports highly misleading when viewed in isolation. It’s a numbers game — looking at aggregate counts & metrics is the best way to tackle ANRs.

The Activity component stands out from other components in that it captures some initial data by calling preDumpIfLockTooSlow() and dumpAnrStateLocked() before it calls the AnrHelper.

This call is throttled to once per 20s and collects a ‘pre-dump’ of information if the ActivityManager/WindowManager monitor is blocked for >1 s. This is required to get an accurate stacktrace if either of these services are in the main thread’s callstack.

The ActivityManagerService dumps the stacktraces and performs CPU sampling, writing to the /data/anr/ directory, which contains a maximum of 64 of the most recent ANRs.

A stack dump has a limit of 20s and captures Java PIDs, followed by native PIDs, followed by an extra PIDs.

This collects information on the window state/app state and caches it in an ActivityManagerService field.

AnrHelper is the point at which the data sources for all Components converge and follow the same codepath. ActivityManagerService effectively delegates all calls to AnrHelper with the exception of some sanity checks that confirm the process exists.

AnrHelper.appNotResponding() is called which adds a Runnable to an AnrConsumerThread. The consumer thread throttles data collection by only collecting the main stacktrace if the system takes >1 minute to collect an ANR.

AnrConsumerThread additionally samples the Binder if an ANR has not occurred within the last 2 minutes, which simply logs out information on Binder transactions.

AnrHelper then calls into ProcessRecordErrorState.appNotResponding().

A lot of the ANR logic happens in ProcessRecordErrorState:

• CPU usage is monitored.
• ANRs are skipped at this point if:
  • The device is shutting down.
  • There is already an ongoing ANR.
  • The app is crashing.
  • The app is being killed.
• A brief entry is written to the system EventLog to record the reason an ANR happened.
• A similar log is written to the FrameworkStatsLog to record the ANR happened. This is used by Perfetto.
• Memory pressure information is appended to the ANR report.
• The ActivityManagerService attempts to dump the stacktraces which are written to the /data/anr/ directory. A few important notes:
  • This can collect more in-depth information than when Activities ‘pre dump’ the ANR trace.
  • There is a timeout of 20000ms for the JVM to create a trace file.
  • This method can return null if creating the ANR trace failed and SIGQUIT will be raised. A native tracing mechanism will capture information for the culprit’s threads instead, and no ApplicationExitInfo will be recorded.
• If the ANR trace file is null then a SIGQUIT is raised which is handled by a native signal handler that blocks the signal for other handlers.
• The ApplicationExitInfo is recorded. The traceInputStream contains a subsection of the full ANR trace file (specifically, it only contains information relevant to the current process).
• The system checks whether the app’s package is still being loaded.
• The ANR report is added to the system dropbox.
• The famous ANR dialog is triggered.

A “silent” ANR happens when a process is in the background and Android doesn’t consider it an interesting process (i.e. it is not showing any UI).

Silent ANRs do record an ANR trace but the collected information is restricted to the current process only. The system immediately kills the process without showing any dialog after the trace has been collected.

A user-perceived ANR is effectively an ANR that a user noticed. Currently this is just an ANR that was triggered via the Input Dispatch detection mechanism. Google Play Console and others likely detect this by searching for ‘Input dispatch timed out’ in the subject string of an ANR trace file.

SignalCatcher::HandleSigquit() handles a SIGQUIT signal which is usually sent from ProcessRecordErrorState. This handler gathers an ANR trace including a dump of native threads.

ActivityManagerService registers a Handler on the system UI thread.

The Handler reacts to an ANR message sent via ProcessErrorState by calling AppErrors.handleShowAnrUi().

AppErrors.handleShowAnrUi() displays the dialog to users. Whether a dialog shows or not is based on various bits of logic:

• Skip showing if it is a background ANR unless the “show background anr” developer option is enabled.
• If an ANR dialog is already showing, skip.
• If a dialog couldn’t be shown, just kill the process immediately.
• If the ANR has stopped blocking, as reported by the AnrController interface. The AnrController interface is also used by the system to perform custom actions when an ANR occurs.

AppNotRespondingDialog responds allows a user to either wait or kill the app.

If the user waits, the ANR state is cleared on the ProcessRecord and the dialog is dismissed.

AppErrors.killAppAtUserRequestLocked() is invoked when the user selects to kill the app, which then kills the process in ProcessRecord.killLocked(), which sends a SIGKILL.

There are a lot of reasons and points during capture where an ANR could be discarded. This can generally be summed up as:

• If an app has a debugger attached (breakpoints pause execution and it would be annoying to get ANRs while debugging).
• If the component triggering the ANR can no longer be found (e.g. it has died or finished its work).
• If the process is aborting/exiting (the process record/thread/component might no longer be found).
• An ANR is already in progress.

The ANR trace file contains a lot of information about the current system state. There are two mechanisms for capturing it: a JVM mechanism that gathers a lot of information, and a native mechanism that is used as a fallback and captures less information.

Silent ANRs only collect information about the current process and skip intensive captures (such as CPU sampling). This is to avoid overwhelming the system.

The Android docs contain a good guide on how to read an ANR trace. To summarize the report may contain info on:

• CPU usage of the top apps using CPU at the time of an ANR.
• Stacktraces of the most important PIDs, native PIDs, and other interesting PIDs.

A subsection of the full ANR trace for historical processes can be accessed via ApplicationExitInfo from Android 11 onwards. To access the full trace it’s necessary to have read access to the /data/anr directory; to access the portion of the trace relevant to the current process, ApplicationExitInfo is sufficient.

The exit info is recorded after the trace is captured. This is fairly straightforward — the trace is gzip compressed and stored in a global circular buffer at ${Environment.getDataDirectory()}/procexitstore.

The trace might be:

• Overwritten by more recent process exits.
• Present if the process recovered from an ANR and then terminated due to some other reason.

Interestingly there are a couple of places that raise SIGQUIT but never show an ANR dialog. These still result in a trace being written to the /data/anr directory.

These don’t count towards Android Vitals metrics but are noted for completeness.

App Slices use a slightly different mechanism for triggering ANRs. The system posts a delayed SIGQUIT runnable so that if a slice does not pin/unpin within 2 seconds an ANR is triggered.

This mechanism is interesting as a dialog is never shown to users and the app is effectively being hosted within Google’s application process.

A finalizer can raise SIGQUIT if it takes longer than 5000ms to finalize any one object. If this timeout occurs it will then throw an uncaught exception.

There are a few widely used approaches for observability. These methods are better than nothing but can often give misleading or unactionable stacktraces when viewed in isolation.

Google Play Console uses the ANR trace file recorded on the device. This guarantees a that each ANR should have a corresponding error report (if a user has enabled diagnostic data), but is fairly primitive as the stacktrace can be inaccurate.

Google Play console also displays ANR KPIs that affect the Play Store ranking.

It’s unclear how exactly this information is sent, but seems probable that Google Play Services scans the /data/anr/ directory and possibly the process exit reasons to calculate the KPIs.

Firebase Crashlytics use the ApplicationExitInfo recorded on the device. This has similar constraints to Google Play’s approach and doesn’t give insight on Android versions below 11 – which tends to be where most ANRs actually occur due to lower device specs! This also complicates the calculation of ‘user-perceived’ ANRs because AEI doesn’t contain the necessary info to calculate this.

ANR Watchdog uses a thread to post a message to the main thread. If the process doesn’t process the message within 5s then it grabs the main thread’s stacktrace.

This approach is prone to false positives and is often too late to see the real issue.

Catching SIGQUIT in a signal handler is a mechanism that allows application to get the stacktrace at the time of an ANR. However, this approach only works for foreground ANRs and suffers the same issues where the stacktrace.

Embrace takes a holistic approach by showing you an ANR flame graph in combination with the AEI approach. This shows you exactly what the main thread was doing in the few seconds before an ANR occurred.

This is achieved with distributed sampling. The SDK uses a monitor thread to check whether the main thread has been blocked for ~1 second, and if so, takes a stacktrace at regular configurable intervals.

On its own this information is not useful. However, our backend will stitch together thousands of similar sessions and constructs a flame graph which shows you exactly which methods were called in the build-up to an ANR.

This approach wins because:

1. ANRs are often caused by multiple slow method calls being invoked sequentially. The flame graph shows you exactly how long each call takes and allows you to address the dominating factor, whereas a stacktrace approach only gives you the last method call
2. It is capable of detecting when the main thread was blocked for a long time (~4.5 seconds) but an ANR didn’t technically trigger.
3. Embrace also captures ApplicationExitInfo and links it to flamegraphs generated from distributed samples. This helps weed out false positives that are a disadvantage of the watchdog approach.

This requires a change of philosophy from “how many ANRs occurred in my app?” to “how many times was the main thread blocked for an unacceptable amount of time and what caused it?”.

If you’d like to see how Embrace can help your mobile team identify and solve ANRs more effectively, you can request a demo here.