Experiments

Experiments and feature flags

info

Experiments and feature flags are available in the Embrace Android SDK 9.3.0 and later. If you are on an earlier version, update the SDK before you integrate.

Most apps run A/B tests and roll features out behind flags, but the results usually live in a different tool than your crash, performance, and session data. Embrace closes that gap: when you tell the SDK which experiments and feature flags a user is exposed to, every session, log, crash, and trace produced from that point on carries that context.

That lets you answer questions like:

• Did the new checkout variant increase the crash rate?
• Is the app slower to start for users in the treatment group?
• Which flag was enabled on the sessions where this error spikes?

How it works

You declare enrollment in experiments and feature flags by calling the Embrace APIs every time your app starts up, as early as possible, so as much of the telemetry from that app instance will be annotated correctly. The Embrace SDK is not an experimentation platform and does not decide which variant a user gets, so it is up to you to do the bookkeeping and update the SDK if experiment and feature flag state changes. In other words, you keep using whatever system you already have for experimentation and pass the resulting assignment to Embrace so your telemetry can be segmented by it.

A few properties are worth knowing before you integrate:

• Experiment state is associated with the app instance. Once declared, an experiment or feature flag is attached to everything the SDK reports for as long as the app runs, irrespective of session boundaries.
• State is not persisted across launches. Cold-starting the app clears the declared state, so you have to declare it again on every start.
• Experiments and feature flags are tracked separately. Each is identified by its kind together with its ID, so an experiment and a feature flag can share the ID (e.g. dark-mode) and stay independent of each other.
• The first call wins. The first call for a given ID fixes its variant and start time. Later calls using that same ID change nothing, even after it has been untracked.
• Untracked experiments and feature flags are kept. Untracking sets an end time rather than removing it, so a variant the user was exposed to earlier in the app instance stays visible in the data.

info

We recommend tracking experiments and feature flags ASAP, ideally before the Embrace SDK starts. Unlike most of the Embrace API, the SDK accepts these calls and constructs the experiment state when it starts, applying it to all telemetry recorded after. To ensure all your data are associated with your experiments, track them before calling Embrace.start().

Tracking experiments

Use trackExperiment for a single experiment, before or after the SDK starts:

class MyApplication : Application() {
    override fun onCreate() {
        super.onCreate()

        Embrace.trackExperiment(
            id = "checkout-redesign",
            variant = "treatment"
        )

        Embrace.start(this)

        Embrace.trackExperiment(
            id = "http3",
            variant = "control"
        )
    }
}

Both variant and startedAt are optional. Omitting startedAt uses the moment of the call, which is what you want when you declare enrollment as soon as you resolve it:

// No variant. You only care that the user is in the experiment.
Embrace.trackExperiment("new-onboarding")

// Enrollment started earlier, so you backdate the enrollment to a timestamp on the device clock.
Embrace.trackExperiment(
    id = "checkout-redesign",
    variant = "treatment",
    startedAt = experimentEnrollmentTs
)

startedAt is a timestamp in milliseconds since the Unix epoch.

When you resolve several assignments at once, track them in a single call with trackExperiments. Build each one with createExperiment:

val experiments = assignments.map { assignment ->
    Embrace.createExperiment(
        id = assignment.key,
        variant = assignment.variant
    )
}

Embrace.trackExperiments(experiments)

info

If you modify experiment and feature flag tracking after the SDK has started, it's recommended to use as few calls as possible, as each call updates the internal state.

Tracking feature flags

Feature flags work exactly like experiments and follow the same rules. They are kept as a separate kind so the dashboard can present them as a distinct experience:

// Tracking a flag that is just enabled, with no variants
Embrace.trackFeatureFlag("new-video-player")

// Tracking a flag with different variants
Embrace.trackFeatureFlag(
    id = "checkout-button",
    variant = "green"
)

// Several at once, with and without variants
Embrace.trackFeatureFlags(
    listOf(
        Embrace.createFeatureFlag("new-video-player"),
        Embrace.createFeatureFlag(
            id = "checkout-button",
            variant = "green"
        ),
        Embrace.createFeatureFlag(
            id = "offline-mode",
            variant = "aggressive-cache"
        )
    )
)

variant is optional on a flag, so use it when the flag selects between named variations rather than just switching a feature on. Leaving it out records the flag as enabled with no variation.

Untracking an experiment or feature flag

When an experiment or feature flag no longer applies for an app instance, untrack it so telemetry from that point forward is annotated correctly:

// A single experiment or feature flag
Embrace.untrackExperiment("checkout-redesign")
Embrace.untrackFeatureFlag("new-video-player")

// Several at once, with an explicit end time
Embrace.untrackExperiments(ids = listOf("checkout-redesign", "new-onboarding"))
Embrace.untrackFeatureFlags(ids = listOf("offline-mode", "dark-mode"), endedAt = userLoggedOutAtMs)

Untracking is scoped to the kind you call it on: untracking the experiment dark-mode leaves a feature flag with the same ID untouched. Only the first call for a given ID takes effect, and IDs that were never tracked are ignored.

You do not need to untrack anything at app exit. The declared state lives only for the current process.

A typical integration

Track the enabled experiments cached locally and applied at app startup as soon as you can. Then, as experiments are enabled and disabled, track and untrack them as soon the changes are applied.

Since re-declaring assignment to an experiment already tracked does nothing, you can safely call trackExperiments without needing to filter by the recently added one. The flip side of this is that you can't update the variant of an experiment that is already tracked for a particular app instance, so beware of that when you are using this API.

class MyApplication : Application() {
    override fun onCreate() {
        super.onCreate()

        // Buffered and replayed at startup, so these apply to the first session of this launch
        Embrace.trackExperiments(
            experimentationClient.cachedAssignments.map { assignment ->
                Embrace.createExperiment(
                    id = assignment.key,
                    variant = assignment.variant
                )
            }
        )

        Embrace.start(this)

        val updateSuccessCallback = { enrolled: List<Assignment>, removedSinceUpdate: List<Assignment> ->
            applyUpdates(enrolled, removedSinceUpdate)
            val updateTs = Embrace.getSdkCurrentTimeMs()

            Embrace.trackExperiments(
                enrolled.map { assignment ->
                    Embrace.createExperiment(
                        id = assignment.key,
                        variant = assignment.variant,
                        startedAt = updateTs
                    )
                }
            )

            Embrace.untrackExperiments(
                ids = removedSinceUpdate.map { assignment -> assignment.key },
                endedAt = updateTs
            )
        }

        experimentationClient.updateAssignments(
            lastUpdate = experimentationClient.lastUpdateTs,
            callback = updateSuccessCallback
        )
    }
}

Limits and validation

The SDK bounds what it accepts so that a misconfigured experimentation setup cannot grow your payloads without limit. The defaults are:

Limit	Default
Experiments/flags per process	500
Identifier length	128
Variant length	128

The per-process limit covers experiments and feature flags together, both those still active and ones you have untracked. Embrace can adjust these limits remotely, within fixed ceilings.

Validation is per entry, so one bad entry never invalidates the rest of a batch:

• Surrounding whitespace is trimmed from IDs and variants.
• An ID that is empty once trimmed is dropped.
• An entry whose ID or variant is over the allowed length is dropped. Values are never truncated, since a truncated ID would refer to a different experiment.
• null, an empty string, and a whitespace-only string all mean the same thing for variant: no variant.
• Duplicate IDs within a single call resolve to the first one.

Invalid and over-the-limit entries are dropped silently without throwing, so it's worth checking your IDs and variants against these limits while you integrate.

Where is experiment data recorded in telemetry?

Tracked experiments and feature flags are attached as an emb.experiments attribute on the session part span and on every log the SDK emits, which is what makes them available for filtering and comparison in the Embrace dashboard. Because the value is also mirrored to storage, a crash report recovered on the next launch still carries what was declared when the crash happened.

If you export your telemetry to another backend, the same emb.experiments attribute travels with the exported spans and logs.