Nightclub Tycoon: Are Idle Games Clicker Games?
Findings
Increased missions results in weakly improved, although statistically insignificant, increases in retention. Ultimately, we did not find strong evidence of the “Cookie Clicker” hypothesis: results were particularly negative when both establishments and missions were 6x.
The remarkable decline in establishment variants could be because establishments involve spending money in the short term, even though that spending will generate future income. This is consistent with hyperbolic discounting, but more research is needed to examine the adverse effects of spending on idle games.
The positive effects of increased missions may have a more significant impact than examined here. Future experiments should look at decreasing the base multiplier to find the height of the experimentation impact curve.
Next steps
xxx
Hypothesis
If completing checklists or goals helps explain idle game popularity, adding more checklists and goals will enhance retention.
Idle games revolve around passively collecting resources and using them to upgrade institutions that generate more income. They contain no “loss” condition; player action is encouraged by time-limited live-op events and limited idle resource collection (players log in to empty resource storage so it may fill again).
As a low cognitive intensity game, our hypothesis proposes genre popularity comprises the following features:
The visual sense of progression.
Within a level, players unlock new worlds and upgrade resource generators.
The audio/visual experience of spending and collecting resources.
Completing checklists or goals.
Our experiment’s design evaluates the impacts on 2. the audio/visual experience of spending and collecting resources and 3. completing checklists or goals.
We’ll massively increase (6x) the number of checklists and missions while holding the economy effect constant.
Game Context
Nightclub Tycoon is an idle game by German developer IterationOne (who are hiring!).
Players visit nightclubs in locations across major cities. As time passes, players earn passive income from nightclub patrons. Players re-invest those earnings to earn even more through various economic strategies. However, the gains of those strategies are daily limited. Once they’ve saved enough, players may purchase a nightclub at other locations and generate further income to upgrade that club’s earning potential.
Player strategy:
Login whenever the passive income collection stops; “empty” storage before limits means resource generation is not collected.
Monetised as expanding storage limits.
Assign characters to income generators that boost production stats.
Monetised as character loot boxes or cards.
Engage in rewarded-ad events (e.g., celebrity visits, external investment) that boost revenue and progression.
Monetised with rewarded-ads.
Wait! Resource generation happens in real-time.
Monetised as time skips.
There’s no “fail” condition; players must simply wait to overcome challenges. The genre traces itself back to at least Cooke Clicker, a 2013 phenomenon in which players tap a cookie to earn points to purchase resource generators, which affect future point production.
The idle progression generated from simple taps becomes something like a Skinner box.
However, idle games limit action-initiated resourced generation; whereas Cookie Clicker points are earned by clicking on the cookie, modern idle games have a finite number of claimable rewards at any one time. The finite clickability reduces retention, as it reduces positive cause-effect loops. However, the connection between idle games and cookie clickers needs more examination. The rise of the “idle arcade” genre revises these questions but requires more active play input.
To test the impact of adding positive collection centres, we significantly ramp up (6x) the number of missions while holding the currency earned constant.
Target Metrics
Primary
Increase D1 retention from 40% to 42% (5% uplift).
Secondary
Increase key milestone progression (% of players building build “Bar Brooklyn,” the location after the Berlin nightclub)
Experiment Design
After completing a task, players earn progress toward a mission goal, and once they’ve satisfied the quantity goal, they claim several hard currency (gems).
Players receive “toasts,” reminding them of their progress as the quantity requirements are progressively satisfied.
The number of missions listed in the Berlin nightclub, the game’s first level, will increase by 6x from 7 to 42 missions. Since the mission requirements are listed in large quantities, it’s easy to divide the totals into smaller requirement amounts and reduce the price by a similar factor. This increases the payout rate (i.e., players receive their paycheck faster), but the amount is constant.
We will increase the number of upgrades listed for each resource generator. Still, similar to the mission design, the progression rate is held constant by dividing the benefits across more levels. Improvements are locations where players spend resources on upgrades to improve currency output.
UI/UX
For each second the button is held, or when currency is claimed for missions, an audio-visual cue plays, and “haptic” feedback (a slight rumble on the player’s device) is triggered. The rate of resources drained from holding a button increases along an exponential curve—the longer a player holds, the faster resources are drained.
Experiment Execution
Control: No experimental treatment.
Variant 1: 6x (from 7 to 42) the number of missions at each Club Level in Berlin.
The mission design and rewards are held constant.
Each mission grows from 1 to 6 “checkpoints” equidistant from one another.
I.e., “Serve 200 guests at the restaurant” becomes six missions (Serve 33, 67, 101, 134, 167, 200 guests at the restaurant)
Variant 2: 6x (from 7 to 42) the number of upgrades at each Club Level resource in Berlin.
The Improvement design and rewards are held constant.
Each upgrade grows from 1 to 6, equidistant from one another.
I.e., the Dazzling light show becomes six upgrades. (Dazzling light show stage 1, Dazzling light show stage 2, etc.)
Implementation Examples
Variant 1: 6x the number of missions
Control, e.g. upgrade the Bar (Rank 2) by 25%:
6x Missions, e.g., upgrade the Bar (Rank 2) by 5%, 11%, 17%, 22%, 29%, etc…
Variant 2: 6x the number of upgrade “improvements”
Control, e.g., seven improvements (box of ponchos, barbed wire, no refunds poster, etc):
6x upgrade improvements, e.g., forty-two improvements (extra muscle, kids walkie-talkies, a box of ponchos, barbed wire, no refunds poster, etc.):
Analysis Plan
We used a one-sided t-test to test for 80% power and 95% statistical significance on D1 retention. The experiment was strictly run on Android.
Power Analysis
Results
All variants failed to show a statistically significant uplift. Only one variant, the increase in mission, saw mostly positive increases.
A 6% increase in players reached the gameplay milestone (41.5% to 44%).
When both improvements and missions were upgraded, players played longer but with fewer sessions, perhaps seeing the missions as “stretch” session goals.
The increase in mission variants results in far higher claim rates from missions overall, even considering the rise in quantity.
Control players claimed 41% of their six missions (~2/6), while both mission variants saw increases in claim rates: variant 1 60% (~21/30), Variant 3 70% (~25/30).
The merged variants performed poorly overall, more so than the combined outcome of missions (positive) and improvements (negative).
We also examined the possibility that players experienced “negative inertia.” Players with 6x mission or improvement might become accustomed to the dopamine cadence and withdraw once it declines. We did not find evidence of this, as nearly all variants churned at similar rates after the experiment.
Conclusions
Increased missions results in weakly improved, although statistically insignificant, increases in retention. Ultimately, we did not find strong evidence of the “Cookie Clicker” hypothesis: results were particularly negative when both improvements and missions were 6x.
The remarkable decline in improvements variants could be because improvements involve spending money in the short term, even though that spending will generate future income. This is consistent with hyperbolic discounting, but more research is needed to examine the adverse effects of spending on idle games.
The positive effects of increased mission may have a more significant impact than examined here. Future experiments should look at decreasing the base multiplier to find the height of the experimentation impact curve.
