OpenTelemetry for dummies reasonably intelligent noobs

As I understand it, OpenTelemetry solves three big longstanding problems in the traditional observability space:

1. Frontend and backend teams don’t speak the same language.
2. Your data is held hostage by your vendor relationships.
3. Observability data isn’t connected to user experience and business outcomes. (This is the one I feel especially strongly about.)

Further down in this post we’ll walk through each of those problems and then look at some specific use cases where OTel really shines. Before we get into that, let’s get this question out of the way…

Simply put, OpenTelemetry is an open standard for collecting and describing what I used to call “performance data” and what I’m now learning to call “telemetry”.

The best analogy I’ve heard is to think of OTel as a universal protocol… kind of like HTTP but for observability. When everyone instruments their software using OTel, the telemetry they produce looks the same, regardless of which tool they’re using or what layer of the stack they’re monitoring.

There are three main signal types:

• Traces – a record of how a request moved through your system
• Metrics – aggregated measurements like error rates or response times
• Logs – timestamped event records

(Do not be fooled by my authoritative-looking definitions. I grabbed those from some of our awesome Embrace docs. I get what metrics are, of course, but I’m still wrapping my head around traces and logs. For today’s purposes, those handwave-y definitions will suffice.)

The important thing to understand is that OTel defines how all three of those signals are collected, named, and exported, so a trace from a mobile app and a trace from a backend service can actually talk to each other. That’s huge!

The success of OTel is built on three principles:

• Transparent – no black-box proprietary formats
• Portable – you can switch tools without re-instrumenting everything
• Extensible data collection – you can add custom signals for your own app’s specific behavior

That sounds dry, so here’s a more exciting way to say it: OpenTelemetry gives you unprecedented freedom to share data across teams and platforms.

Now let’s talk about how that freedom helps you solve three common problems…

I don’t love new technology just because it’s new. I’m not that species of nerd. I’m the species that likes to solve human problems, so the best place to start is by understanding the problem that any new technology is trying to solve. Here’s my understanding of the history that led up to the current problem.

As I’m sure you already know (but bear with me), modern software runs on two layers:

• Server-side – the databases, APIs, and infrastructure that your backend and SRE teams watch over
• Client-side – the mobile apps and browsers where real users actually experience your product, which your frontend performance team typically watches over

Both sides generate data about how things are performing. The trouble is, these are usually entirely separate tools that don’t share a common set of telemetry and don’t connect in any meaningful way. Because of this, engineering teams aren’t able to speak the same language.

The result is a mess. What a surprise.

When something goes wrong – say, a checkout flow starts failing – here’s what different folks see:

• Your backend team sees a spike in server errors.
• Your mobile team sees a crash signal.
• Your frontend web team sees that Interaction to Next Paint is hurting.

Each team is working from different datasets, using different tools and completely different mental models. That’s a huge disconnect!

Everyone is trying to connect dots they realistically can’t be expected to connect, and it’s not fair to expect them to. Site reliability teams are forced to guesstimate user and business impact from service and infrastructure metrics. Developer teams are tasked with magically solving issues that are out of their control. And so on.

One of the widely touted advantages of OpenTelemetry is that it connects your whole stack. Great, I guess?

But TBH “connect your whole stack” is one of those expressions that doesn’t land for me. (Like I said, I’m not that species of nerd.) The real win is that OTel connects your teams. In doing that, it helps everyone work together to help your users. And ultimately, helping your users helps your business.

OTel isn’t about tech for the sake of tech. It’s about people. (There’s a Soylent Green joke in there somewhere, I know it.) I care about people, and that’s why I think OTel is important.

Before OTel, every observability vendor had their own agent, their own data format, and their own proprietary schema. Traditional vendors favored closed ecosystems and opaque pricing.

That sounds bad because it is bad, but for a long time no one called it bad because that was just the way the tools worked. (To be clear, I’m not accusing any vendors of unsavory intent. They were just following standard practice.)

If you wanted to switch vendors, you had to rip out your instrumentation and start over. Ugh. Not only was that hideously expensive, but even if you exported your data, there was always the risk of losing or compromising it.

OpenTelemetry means your data is no longer held hostage by your vendor relationships.

Your team doesn’t have to re-instrument when changing vendors. You write your instrumentation once and it works with any OTel-compatible backend – Honeycomb, Grafana, or whatever you choose.

I saved my favorite topic for last. If you care about your users and your business (and please, I hope you do), this is for you.

Traditional observability tools were built to watch servers, not people. They can tell you that a service had increased latency at 2pm, but they can’t tell you that this was the fallout:

• 12,000 users experienced a frozen checkout button
• Android users suffered the most
• Your conversion rate dropped by 6%

That’s because user experiences are messy in ways that server logs aren’t. A backend request takes milliseconds. A user session can take minutes, or even hours, winding through a dozen interactions on a device you’ve never tested on a network you can’t control.

OTel connects your servers, your mobile apps, and your web frontend into a single shared view. When that view also includes user engagement and business metrics, then you get the full, glorious technicolor picture. It’s the difference between monitoring your infrastructure and understanding your users.

I’ve covered these already as problems, so now let’s reframe them as solutions. As you consider these, it becomes pretty clear that these use cases all build on each other: from communication to diagnosis to prioritization based on business impact. It’s a beautiful thing.

1. Break down team silos

Frontend and backend teams need to speak the same language and access the same datasets to draw insights from user experiences. When both teams’ tools are built on OTel, a mobile engineer and an SRE can open the same trace, look at the same data, and have a productive conversation about what broke and where.

2. Diagnose an issue across the stack

Without OTel, a backend engineer investigating a high error rate has no visibility into what users were actually doing when those errors hit. With OTel-connected frontend data flowing into the same system, they can see that the errors coincided with a new app version rollout, which affected users on a specific device type. The root cause becomes findable.

3. Prioritize what to fix based on user impact

We all want to prioritize issues by understanding actual user impact, which is only possible by connecting backend observability data directly to end-user experiences. OTel makes that connection possible. Not every p99 latency spike is worth an all-hands page, but if that spike is hitting your highest-value users during checkout, it absolutely is!

4. Switch vendors without starting over

Because OTel is an open standard, your instrumentation isn’t tied to any particular vendor. If you switch tools, your data comes with you, so you’re not starting from scratch.

Plumbing may not be cool, but we all know what happens when it doesn’t work. In this case, it’s the plumbing that connects what’s happening in your servers to what’s happening for your users. That’s a gap that’s existed for a long time. If you want to be a hero in your organization, grab your wrench and close that gap.

• Why Embrace is leading the way in OpenTelemetry for mobile
• Embrace and Honeycomb: Connecting user experience to backend truth
• Send your mobile and web telemetry wherever you need it