In our webinar, Lucas Klegen, the Manager of Engineering Operations from Wildlife Studios, talks about the challenges top mobile gaming studios face. These include issues around scalability, resources, observability, and user support.

In his talk, he discusses five case studies spanning four different mobile games — Sniper 3D, Sky Warriors, Color by Number, and Tennis Clash. These case studies illustrate how Embrace has helped Wildlife:

  • Keep their mobile games below bad behavior thresholds.
  • Become featured more frequently in the App Store and Google Play Store.
  • Save valuable developer time.
  • Anticipate app issues before they affect users.

Interested in learning more about Embrace? If you’d like to see how complete mobile visibility can revolutionize how you deliver incredible mobile experiences, get started with a free trial today.

You can also watch the on-demand webinar and read the full transcript below.

Presented by Colin Contreary and Lucas Klegen.

Video length: 38:01

Date premiered: Mar. 30, 2023

Lucas Klegen: Thank you very much, Colin. Hello everybody! Colin, can you hear me well? Just to check on the sound. That it’s okay?

Colin Contreary: Yes! [Thumbs up]

Lucas Klegen: Perfect, perfect. So, hello everybody! I’m very happy to be here. Thank you for inviting me, Colin. It’s a true pleasure. So, before we start with the presentation and I talk about how Wildlife uses Embrace to improve its technical performance, I would like to briefly introduce myself so that we can get to know each other a little bit better.

So, as Colin mentioned, my name is Lucas Klegen. I am currently a Manager of Engineering Operations here at Wildlife Studios. Just a little bit about my background — so I’m an industrial engineer from the Polytechnic School of the University of São Paulo. I worked as a Management Consultant for over three years and for the past two years, I’ve been a manager here at Wildlife Studios.

During my time here at Wildlife Studios, in these past two years, I’ve done a lot… thankfully [laughing]. I’ve worked on multiple strategic technical initiatives, and one of the main initiatives that I’ve worked on with all of our game teams is on technical performance. And this is how I got to know Embrace, and this is how Embrace and Wildlife have been working together for the past… I’d say it’s one and a half… almost two years now. Right, Colin?

Colin Contreary: [Nods “yes” and smiles]

Lucas Klegen: So this is just a little bit about me.

So, I would like to go through the agenda for today. I’ll start [by] speaking a little bit about Wildlife Studios. I imagine that some here may or may not know what we do or how big we are, so I think that I’ll talk a little bit about the company [to] help bring in some context and speak to the importance of this presentation. Then I’ll go over challenges that Wildlife Studios faces as a large mobile gaming studio. Here, we’ll [have] more of a focus, of course, in technical challenges that we face. The next three items are related to five cases that I’m going to talk about that really show how Embrace concretely helped Wildlife. In the first two cases, we’re going to talk about how Embrace helps keep games below bad behavior thresholds, then how Embrace helps us leverage best-in-class insights to focus on actual issues and not false alarms, and, finally, there is a last case that is going to show how Embrace concretely helps us to anticipate issues before we have to hear about them from our player base.

So these are the contents for today and I really do hope you enjoy!

So, what is Wildlife, what do we do, and, well, who are we? We are a Brazilian mobile gaming studio founded in 2011. We’ve been around close to 12 years now. We know our mission is to connect the world with the fun of gaming. We really strive to elevate and liberate creators so that we can create games that are fun and that help people, engage, learn, and interact — that’s our mission.

I brought here some success metrics that, you know, really paint a picture of how big we are and what we aim to achieve. We’ve launched over 60 games and not all these 60 games are what we call “live” games, but we have about eight or nine live games which are games that we constantly launch new content and updates throughout these. [In the] almost 12 years that Wildlife exists, we’ve launched over 60 games. All these games have accumulated over 3 billion downloads. I’m always impressed by that number because 3 billion is [just] less than half of the [world] population, if you think about it, so it’s really interesting to see, you know, how big the reach that our games have.

And I also brought some notable partnerships, really focused on one of these specific games that is featured here — Tennis Clash. Which is, as the name says, a tennis game. So we’ve done a lot of partnerships with famous and really important championships. I mean, if you’re a tennis fan, you definitely know these. So, Roland Garros, the U.S. Open, the Billie Jean King Cup, we’ve had partnerships with all of these tournaments. Basically, what that means is that while these tournaments are actually happening in real life, we also have a virtual gaming tournament inside the game which players can compete and play. In one specific edition of the Roland Garros partnership, the winner of that tournament inside Tennis Clash was awarded a trip, fully paid, to watch the Roland Garros final. So that was really fun — these are just some of the partnerships that we have. This is a little bit about Wildlife, what we do, who we are, and what’s our size — just to set the background and set the context for what comes next.

Here I have, you know, I brought a few screenshots of some of our games. As you can see, we’re really genre agnostic, so we have games ranging from medieval games, coloring games (which I’m going to talk about), fighter jets playing (which I’ll also talk about), and shooting games. So, we’re really genre agnostic and what really matters, as I mentioned, is that we liberate the creators that we have here to create whatever mobile games that they think [are] fun. That will help people engage, interact, and learn together. So, that’s a little bit about Wildlife.

Wonderful! On to our next point of the agenda — what challenges does Wildlife face?

I’m going to start with the first one here. First thing’s first — scalability. We consistently launch new games and whenever we’re launching a new game, or in the first few weeks that a game is launched, scaling the game and scaling the game safely is challenging. There’s a lot of uncertainty in the size of new games [concerning] how big is the game going to get. That’s something really hard to estimate. In a game’s life cycle, we have the alpha, beta, soft launch, and global launch. Before the global launch, we do have some insights on how big the game can grow, but when we go global, that’s when things really scale and that’s when things can be very unpredictable.

So how do we provision infrastructure to safely balance the size that the game needs to get and how do we balance this with cost? Because, as we scale infra[structure], costs also increase. That’s the type of challenge that, particularly when we’re launching new games and growing really new games, it’s something that we face.

Resources — I think that every company in the world faces a challenge of allocating and investing finite resources. This is not different for Wildlife. Part of this challenge involves deciding how we’re going to allocate valuable developer time. So… are we going to launch new content? Are we going to improve existing features in the game? Are we going to allocate the developer to work on issues, on support tickets, on bugs? Deciding that is something that is a challenge for, I believe, every tech company — it’s no different for Wildlife. Especially when we’re talking about developer time, which is a very, very, very precious resource.

Observability — as a mobile gaming company, we generate a lot of data. So we track: player behavior, how they are progressing in the game, monetization metrics… Data on its own doesn’t mean anything, but organizing the data and extracting valuable information is something that is super challenging. When we talk about technical performance data, a great example is when we have ANR incidents or crash incidents, which I’m going to talk about a little bit better soon, [we ask] how do we look at all this these stack traces, and how do we look at all this data, and extract any valuable information from it, and extract actionable information from it.

And finally, support. So, we pride ourselves on being close to our player community and offering them the best support there is, but in an ideal world, we would never have to support requests because we’d anticipate every possible issue. Anticipating issues and acting swiftly before these issues escalate is something that’s also super important.

A great example that I can give is mitigating risks when launching new games. It’s super important for us to cause a positive impression and drive momentum when we’re launching new games. This is a challenge that we face and something that we work on. Especially because tech performance and game launches can be problematic, as I’ve already mentioned.

We’ve seen in several recent launches in the market in other companies how when you launch a new game and technical performance is an issue, this can create very bad publicity. [This really does] not help the game gain popularity. Mitigating these risks in this very crucial phase is super, super important.

With all this in mind, we decided to partner with Embrace because we felt that Embrace could help us with many of the challenges that I’ve mentioned that Wildlife faces. We did assess several providers, and we did run a pilot with several different providers, but ultimately, we realized that Embrace was the best partner for us for several reasons. Some of which I already mentioned and [some of which] I will go into more detail.

Embrace helps us continue to keep our games consistently below bad behavior thresholds. Embrace provides us with reliable and very deep data and insights based on the technical performance of our games. Embrace helps us mitigate issues quickly before they reach our player base and, overall, Embrace helps us have healthier apps so that, as I mentioned, Wildlife can continue to focus on its mission of liberating and elevating creators to great games that will help the world engage, learn, and interact together. This is why we partner with Embrace and this is the origin of our partnership.

So, as I mentioned, there are five case studies related to four games. Sniper 3D, Color by Number, Tennis Clash, and Sky Warriors, that I will go over in the following sections. As I mentioned, these are all live games. So these are all games that we launch content on a weekly basis. They are all games that are very important for the company’s revenue and, as I mentioned, Embrace helped us with the tech and performance in each of these games. [They’ve] really helped us show, in concrete terms, the value that Embrace creates for Wildlife.

The first topic that I want to go into is: how Embrace helps Wildlife, and one of the games here, specifically, stay below bad behavior thresholds consistently.

Before I get started into this specific case, I think it’s really, really important for me to give a bit of context because perhaps not everybody here is familiar to what I mean by bad behavior thresholds. What does that mean? What’s the impact of that? I think it’s important for me to give a little bit of context here of what about this.

Both Google and Apple — they monitor the frequency in which any app actually, not only mobile games, but any apps that are downloaded in their store — the frequency in which these apps have issues and fail. When I say fail, I mean specifically when they have crashes or ANRs from a user’s perspective. ANR, which means “Application Not Responding”, is kind of the same thing as a crash because the app closes. The user is unable to continue playing or to continue using the app. Gameplay is interrupted and it’s a problem for players. In technical terms, they are different things, but I don’t think we have to get into that detail. Both Google and Apple track the frequency in which this happens to the apps that are downloaded in their stores. [At] Google, they do disclose what are these thresholds [and] what are the limits for each one of these issues.

On the table here on the left hand side, you can see these percentages. For crash rates, that’s 1.09% and for ANR rates, that’s 0.47% overall across all devices. Per specific mobile phone type, it’s 8% percent. So these percentages, they seem kind of low, but they’re not. Especially when we look at the overall thresholds. Because, let’s think of it this way, anybody in the whole world can download an app in the Google Play store or the App Store. So that means that anybody that has either the best device in the world, like the latest Samsung Galaxy or the latest iPhone to people who have really, really low-end devices. When we look at everybody that can download the game and all the possible devices in the world, there are a lot of devices that are more prone to fail because they have low memory because they have low specs. So when we look at all the devices, these percentages, they aren’t actually quite low, they’re actually quite challenging to achieve. You know, 1% means that 1 out of every 100 devices can fail no more than that — [they] can have ANR and crash issues no more than that, and for ANR’s it’s one over 200. So, I’m just trying to depict how challenging it is actually to stay below these thresholds.

iOS doesn’t disclose the exact percentages, but we know that if you’re above a certain number empirically, if you’re above a certain number, that can cause issues with your organic distribution in the App Store.

So, as I mentioned, why?

Why does Google and Apple track the frequency in which an app crashes or has ANR situations? Because they don’t want to promote apps that fail, apps that have tech and performance issues, they want to promote healthy apps so that people can continue downloading apps in the store and returning to the store. That’s why they track these issues and that’s why they, in a certain way, penalize developers who have a lot of crashes and ANRs, and that’s why they promote developers and companies that have healthy apps.

On the right hand side, you can see this is just an illustrative print screen from the Google Play Console where you can track what’s the percentage for your specific app.

So as I mentioned, Apple and Google, they want to promote healthy apps and, in a certain way, they don’t want to promote, I wouldn’t say penalize, but they don’t want to promote apps that are not healthy that will cause a lot of issues for users.

So when I say promote, what do I mean? Basically, there are a few ways that Google and Apple promote apps in the store. On this screen over here, you can see this is the ranking chart. One of the ways that Google and Apple can promote apps inside their store is by ranking them higher. I know that when I want to download a new game or a new app, I usually look at the top charts. So if you’re among the top 10, top 15, the chance of being downloaded is higher. Be advised, technical performance is not the only parameter that Apple and Google take into consideration to rank apps, but it is one of those.

And another way that Apple and Google promote different healthy apps and apps that are below the threshold is by featuring them in a specific section in the App Store. Here you can see that sometimes it will promote new updated games that are healthy or just feature and highlight games for other reasons. For an app to qualify and be promoted, it’s important, it’s crucial I’d say, to be below these thresholds. [I’m] just setting this stage and just giving some context on why these thresholds are so important and why do developers strive to be below the thresholds. And, you know, in business terms, this means that if you’re featured more frequently your potential for organic distribution increases, more people will organically download your app, more people will organically install your app, and play and use your application.

In a marketing perspective, this is the potential and the importance of being below the thresholds and, overall, if you have a healthy app, if you have an app that doesn’t frequently crash or go into an ANR state, you deliver a better experience for the player. You deliver a better gameplay experience and this helps players to stay within the game to help increase retention. This helps increase monetization and, overall, it just helps the business. This is the importance of not only being below the thresholds but having healthy apps in terms of technical performance as a whole.

Now on to our first game that we’re going to talk about — Sniper 3D. I’m going to play a short clip here.

[Video of Sniper 3D gameplay begins]

I don’t think, I, I think that unfortunately; you don’t… it’s not possible for you to listen to the audio. There’s like a cool soundtrack going on but, a fun fact, an interesting fact, Sniper 3D was the first game that Wildlife developed that is 3D, as the name says. So that’s why it’s called Sniper 3D. And it’s super fun. It’s one of our, one of the oldest games, actually, that we’ve developed. I think it’s about seven years old so, yeah, this is just a short gameplay video, but let’s talk about the case.

So, as I mentioned, one of the biggest challenges that we’ve had that we had for Sniper 3D was to keep Sniper 3D below the bad behavior thresholds. It’s something that we invested a lot of developer time in. It’s something that we did — invested a lot of effort to stay below these thresholds. Especially when we launched new content, new features, new maps, that would always require like right after we launched a new content. We would have to devote a lot of time to keep the app below the thresholds and one of the reasons is that, Sniper, as I mentioned, Sniper 3D is one of the oldest games at Wildlife. It’s been around for a while and many people that worked on the game you know are no longer working on this game. It was built using different standards, so it’s really a game that’s complicated to manage complicated to continue building on it and that’s why we have this problem of having to employ a lot of developer time and resources to keep the game below the bad behavior thresholds.

With this, we bought Embrace and Embrace helped us with a very important investigation. Through Embrace’s information and dashboards we were able to see that, we were able to see the ANR distribution of, you know, in our situations the sessions that had ANRs and we realized that 60% of the situations in which an ANR happened the game was on pause, 30% was when the game was in web views, and 10% was others. So obviously we focused our investigation when the game was on pause. So this was a first, you know, important insight that we had through Embrace’s dashboard and information. And, you know, unpause situations are situations in which, for example, analytics calls are made and files are written, or when an app switches activities, for example, to show ads, or when simply the app goes to the foreground. We had a very strong hunch that this was related to showing ads, we ran a few hypothesis tests specifically focused on ads, and we reached the conclusion that we had to disable ads for a subset of devices and, for another subset of devices, stop supporting the game all together.

Here, Embrace also helped us because, through Embrace, we were able to see what was the profile of the devices that were most responsible for causing ANRs and we were able to estimate, ‘Okay, so if we stop supporting these very low-end devices, how far do we have to go in so that we can more easily stay below the thresholds? Do we have to go this far? That far?’ Embrace really showed this profile of devices and we were able to make an informed decision and now we’re more easily below the thresholds. You know Embrace, I mean, Embrace was not the “silver bullet” here, but mainly because there were no silver bullets. But, as I mentioned, Embrace did provide all the tooling so that we could quickly investigate what had to be investigated and so that we could quickly resort to a more drastic solution — let’s say to stop supporting a certain subset of devices. I’m pretty sure that if we had not used Embrace for this investigation, we would probably have spent a lot more time chasing dead ends and chasing false leads instead of investigating what had to be investigated, reaching a conclusion, and making a business decision. This is now how important Embrace was in this situation for us.

Okay. On to the—

Colin Contreary: Oh, Lucas! Lucas, what happened in terms of being featured?

Lucas Klegen: Oh! Of course, of course. Yeah, yeah, of course. So, as I mentioned, once you’re below bad behavior thresholds that really helps a lot with app promotion within the app stores and, for Sniper 3D, we saw a very clear increase in the number of features that we had following the period in which we were consistently below bad behavior threshold. So, naturally, this helped increase organic distribution and, as I mentioned, overall deliver a better gameplay experience for our players. So the tech performance results translated to business results.

Thank you very much, Colin. I was really really excited to show the Sky Warriors video and I forgot to mention this last piece of detail.

Colin Contreary: No worries, I’m here — I’m just here to help.

Lucas Klegen: Wonderful! So, let’s watch the clip for Sky Warriors. The cinematics is really nice for this game. This game is really, really beautiful.

[Video of Sky Warrior gameplay begins]

Colin Contreary: Looks just as good as Top Gun: Maverick — the graphics.

Lucas Klegen: Yeah! I just love it and I’m a huge fan of fighter jets so I’ve really, really enjoyed playing this game. Give it a try! Download it! Yeah, wonderful.

So, here we’re talking about the same challenge, but it’s a different situation for a different game. So, we’re talking about, you know, in contrast, opposed to Sniper 3D, which is one of our oldest games, we’re talking about a newer game. A game that was launched at the end of 2021 called Sky Warriors. And I’m here to talk about how Embrace helped this new game launch stay below the bad behavior thresholds which I mentioned is so important.

So, as I mentioned the problem — the challenge here was to stay below these thresholds for a new game launch and what we saw happening in the first few weeks after we launched the game was that there were generic errors that appeared in other crash reporting tools. We also use a few other tools that we always used before bringing in Embrace and what happened was that, for these other tools, we were seeing some generic issues, but the engineers at Sky Warriors, they were having a really hard time understanding what’s the origin… what’s the root cause [of] this error that we were seeing. So we used Embrace to investigate and to try to pinpoint where the issue was and Embrace showed that the error was related to a chat implementation that was leading to these crashes. So Sky Warriors has a chat feature, a chat capability, in which you can chat with other players and the library for this chat that’s implemented within the game was causing a few issues and Embrace showed us exactly where the issue was happening. So, our crash rate was at 1.2% which was above the thresholds, as already presented, and, as I mentioned, Embrace showed us exactly where the issue was happening, exactly the line of code that caused a crash, and this helped us to quickly fix the issue in a matter of minutes and launch a patch to fix the game. And, with this, the crash rate decreased from 1.2% to 0.3% which figures below the bad behavior threshold.

This has also helped increase our features in the Google Play Store, helped highlight the game for the public, and, as I mentioned, this is super important when we’re launching new games and it made a huge difference for the game to scale really early on in the first few weeks that it was launched.

So, besides, of course, allowing people to send messages to each other and to play the game without interruptions, this was the result and the impact of Embrace in this specific situation.

Wonderful.

So now I’m going to talk about two cases for a different category of problem which is leveraging best-in-class insights to focus on actual issues and not false alarms. So, as I mentioned, developers’ time is super precious, and really reserving developers to work on what matters and what will bring results is super important for any tech company.

I’m going to talk about how Embrace helped us do this and improve this. The game that’s featured is Color by Number — a coloring game. So, let’s see a short clip of this game.

[Video of Color by Number gameplay begins]

This game is super fun. It’s as the name says, a coloring game so you can paint like a preset of pictures, you paint for different numbers, different colors. You can also upload your pictures, and paint up a picture that you found on the internet or a drawing that you drew. So… it’s super fun, it’s really good to relax, it’s really good to, a really good, stress relief. I also recommend downloading it and giving it a try!

Colin Contreary: That last one looked like it was out of The Hobbit!

Lucas Klegen: It was! It was.

Colin Contreary: Oh, yeah?

Lucas Klegen: All right, wonderful. So, what was the issue here? So as I did mention, before we had Embrace, we were using a few other technical performance monitoring tools and we usually saw a lot of data inconsistency and information inconsistency between these different tools and these different sources. And that’s actually one of the reasons that prompted us to look for another solution and ultimately bring Embrace [in] to work with us. So, in this specific situation for Color by Number, we’re also seeing, our different, other monitoring tools were showing different types of errors and it was really hard to reach a conclusion. And in this specific situation, there was a crash that only showed up at Embrace. In this specific situation, it was not showing on other monitoring tools which was worrying.

So this crash was specifically related to an ad network bug and it was impacting hundreds of  thousands of players. And what’s alarming here for me is that, had we not been using Embrace, most likely this bug and this issue would have kept on impacting players for days or perhaps weeks. And, because we had Embrace, we were able to identify, ‘Okay this is an actual bug — let’s go there and fix it.’ And we were able to, as I mentioned, mitigate the impact that we had in our player base. So this was super important and it really helped us investigate what was the right thing to investigate and quickly solve an issue that was being very harmful for the app.

So, as I mentioned. the game team easily solved the issue with data from Embrace and we’re able to deliver a better player experience for everybody who was playing Color by Number.

The next situation that we have is kind of the flip side of this. So, the problem that I’m going to talk about is wasting developer time on false alarms. So, for the same game, what was happening is that another monitoring tool that we used also pointed to a bug in Color by Number. So the game developers, they were scrambling to try to identify what’s happening — this was impacting hundreds of thousands of players. What’s the matter? When we looked at Embrace’s error morning tool, we did not find this issue. So we thought, okay so one of those has to be right and one of those has to be wrong, and what we found out was that the issue that was being reported in the other tool was actually an error from the crash reporting tool libs itself. So it was actually an issue with the library that makes that other error monitoring tool run. Which means that it’s not an actual issue from Color by Number. So, the alarming thing here for me is that if we weren’t using Embrace, we would have spent a lot more time investigating an issue that actually did not exist and because Embrace showed us that, wow, well we’re not seeing the issue. This doesn’t appear to be an issue, we were able to stop all investigations, stop wasting developer time, and say, ‘Okay, yeah this is not an issue. Let’s stop worrying about this and let’s work on something that’s useful.’ So this is another very important, very important, situation in which Embrace helped us.

Wonderful. And for the last topic, as I mentioned, it’s about how Embrace helps us anticipate issues before we hear them from our player base.

[Video of Tennis Clash gameplay begins]

So, the game that is featured is Tennis Clash. A confession I have to make is that this is my favorite game from Wildlife. Like, if you’re a tennis fan or if you enjoy to play tennis, the gameplay of this game is wonderful. It really mimics an actual tennis match and… it’s just… an amazing gameplay. The arenas are super fun, the characters are amazing, so I also invite you to download the game and give it a try.

Okay so the situation here is about how Embrace helped us reduce the issue resolution time in Tennis Clash so, as I mentioned, one of the things that we strive to do is to respond to issues quickly before they impact a large number of players. And, as I mentioned, you know Tennis Clash is one of our live games, so we update the game frequently. We send, we launch new content, we create new arenas, create new players. And, when we do this, when we launch a big update, sometimes regressions can happen. We have to do rollbacks in order to work on the update and then send it out again. So what we want to do is to avoid the impact that perhaps faulty updates can have in the player base, and Embrace’s alert feature really helps us with that.

Embrace’s alerts help us understand when there’s an abrupt increase in errors or crashes and, in this specific situation, we launched a new update for Tennis Clash and we saw quickly, Embrace’s alert monitor, quickly pointed to a spike in crashes and because of that we were able to roll back the changes. And that only impacted a really small part of the player base and we were able to fix the update and send it out again. So, if we didn’t have these really quick and fast alerts, what could most likely happen would be that we would launch the update, the faulty update would have impacted a lot of players, players would hear about it from the player community, and then we would roll back and work and fix it.

From a mobile gaming studio perspective, that’s very bad but with Embrace and with the alerts we’re able to quickly understand when there’s an issue with any sort of update, roll back fix it, and send it out again. So we can be more proactive and respond to issues before we hear about them from players and this is gold. This is super, super important for a company that has so many players playing our games.

Wonderful. So, we’re at the conclusion of our presentation and I just want to go through a few key takeaways of what we’ve spoken about so far. So, as I mentioned, it’s super, super important for gaming studios like Wildlife Studios to continually deliver great player experience to players and doing so requires actionable data — requires very good data. And we chose Embrace because we believe that Embrace can help us maximize the success of our games because we can have state-of-the-art technical performance. As I mentioned, we can now keep games consistently below bad behavior thresholds and this is super important because this helps us be featured, this helps us increase our organic distribution on app stores, and deliver great experience for players [so] we can focus the time of our developers — the precious time of our developers — to work on actual issues instead of false alarms. And, as I mentioned, we can have a better relationship with our player community because we can proactively respond to issues instead of waiting for our player community to be affected and hear from them when issues occur so we can be proactive, and we can work quicker to identify issues and solve them.

End of transcript.

If you’re interested in learning more about Embrace after this webinar, you can start for free today.