How to start experimenting
From King to TapNation via the legendary Lawson at Heseri Games, we’ve worked in experimentation for over a decade. Over that time we’ve consistently seen how, when applied correctly, experimentation can unlock huge growth for games of any size. This article summarises what we’ve learnt across these experiences.
The starting point has been consistent: A team has been working on a game for a while. Performance isn’t where they would like (maybe it’s fallen off, maybe it was never there). They’ve tried a bunch of strategies to turn things around, but nothing has had the desired effect.
Believing in their game, they’ve got in touch. Which is perfect, because this is a great use case for experimentation.
Summarising learnings across every project we’ve worked on, this is our advice if you’re in a similar spot and thinking of trying it for yourself.
1, Operate within a structure:
Experimentation isn’t about taking wild bets and hoping for the best. You should be clear about what you’re trying to achieve and why you think it will work with your player base, then methodically work towards that goal. This means being clear on the metric you’re trying to move and what your biggest hypotheses are.
What metric do you want to move?
Money, obviously. But we have to go deeper than that and carefully consider the implications of each candidate.
It pays to target a level you can impact directly. Let’s consider Average Revenue per User (ARPU). This could be achieved by filling your game with ads. While ARPU may grow, you’ll probably also kill retention, which isn’t what anyone wants.
What about Average Revenue per Paying User (ARPPU)? This might reduce the retention risk, as you’re targeting players who have spent already. But it also means a small target, making it harder to measure reliably. It’s also a really precious segment, one you need to be careful with.
Neither sound like a great option. But they’re not necessarily wrong. The point is to carefully consider the implications of your decision.
We’ve found the most common starting point is early retention. Often day 1 (D1) to day 3 (D3) retention is low, which translates to low retention across the curve. It also impacts the ability to spend profitability on user acquisition (UA).
Working on D1 to D3 has a number of benefits:
Fast answers:
Maximum players exposed to the treatment
The treatment is visible early in the experience by definition
Potential for trickle-down effects:
Lifting D1 to D3 retention can lift later parts of the curve
Learnings could be applied to later parts of the experience
Relatively low risk:
These players are fresh to the game, there is less risk if things going wrong
You may be able to test in cheaper markets, again protecting your best assets
As a side note, it’s often been the case that we’ve started by removing things, balancing levels or reordering when features are introduced. So, relatively light work, which is also helpful.
What’s your strongest hypothesis?
This is so important. Don’t be tempted by anything other than your best theory about what’s going to move your target metric:
If you’re right:
It will be easy to see, players will respond quickly and clearly
Victorious, you can immediately dig deeper into your understanding, testing ways to make the most of it
If not you will have saved yourself a lot of time and energy
This might sound counterintuitive, but a negative result is amazing
You’ve removed a misunderstanding that would have been your biggest blocker to progress
You can now move onto to your next biggest hypothesis, until you find what truly moves your players
2, Maintain a healthy tension between data and creativity:
The best teams balance data and creativity to generate fresh ideas that are grounded in an understanding of their player base.
Focusing too much on data leads to ever-decreasing gains. Teams are inwardly focussed. New features are variations on a theme. They will never be enough to prompt a significant change in trajectory.
Unbridled creativity reduces your roadmap to a lottery. Features with no reference to player behaviour are unlikely to work because they have no basis in truth. Creativity needs to be within the frame of your player base to be effective.
Thinking broadly in a way that relates to user data balances these two needs.
Take inspiration from other games, other apps, the real world, an experiment you’ve read about, anywhere… but tie it back to the understanding (or beliefs if you’re really starting from scratch) you have about your players.
There is no harm in taking inspiration from other experiences, as long as you have a clear reason for doing so. Your goal is to learn, so if you think your players would respond to a competitive, short term goal and you know a live op that has a great one, try it and see what happens.
The true value is in understanding whether or not your players responded to the treatment. If they did, you can start to innovate on it yourself.
To ensure your experiments are based on data, make it a requirement to add an explanation to the hypothesis. For example:
“A live op where players race to complete the next five levels will increase session length because experiment X showed our players love to compete”.
This simple change will help filter out fun ideas that, even if successful, won’t help you build your understanding.
3, move fast
And finally, move fast, test often.
This is the biggest differentiator between teams we’ve seen make progress with experimentation and those who haven’t.
The true value in experimentation is building an understanding of your players.
The more chances you have to learn, the smarter you get and the better decisions you make. That is your competitive advantage.
So be concerned about experiments that are going to take a month or more to ship. Home runs are rare. Spending four weeks building and two weeks (plus) waiting for data for a flat or unclear result is really demotivating (this is another reason for starting with your biggest hypothesis, it is most likely the result has value).
You should aim to get your first swing out the door in a week or two. And keep it simple in terms of cells (control and one test cell only), so you get a result in a matter of days. Review the data, discuss what you think it means and agree which experiment you think will have the best impact next.
Each time you’ll learn something new and make a step forward.
We’ve found getting results back is a real moment for teams. Having data allows the discussion to move on from hypotheticals and hazy understandings. It democratises debate across the team, taking it from “what if” discussions that can be dominated by opinionated individuals, to creativity bound in understanding, everyone can contribute to.
It’s an exciting moment.
Summary:
I work in experimentation because I love it. Every experiment is fun. I see that in the teams we work with. The outcome is rarely as you expect, but the learning always takes us forward.
So, if you want to give it a go, remember:
Operate within a structure
Maintain a healthy tension between data and creativity
Move fast
If you’d like to know more, just get in touch.
Have fun!