Insights on Improving LTV

Item not bookmarked
Resource bookmarked
Bookmarking...
⛏️
Guest Miner:
Sylvain Gauchet
Review star
Review star
Review star
Review star
Review star
💎  x
20

Sasha MacKinnon (CEO of Mino Games), Josh Chandley (COO of WildCard Games) and Eugene Joannides (COO & Co-founder of Mistplay - Loyalty Program for Mobile Gamers) discuss how to approach improving LTV through the lenses of product, data and marketing.

Source:
Insights on Improving LTV
(no direct link to watch/listen)
(direct link to watch/listen)
Type:
Panel
Publication date:
October 7, 2020
Added to the Vault on:
October 25, 2020
Invite a guest
These insights were shared through the free Growth Gems newsletter.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
💎 #
1

The high level definition of LTV as ARPDAU*Retention has no application in the real world: changes in the game can have unforeseen side effects. You can improve ARPDAU but a few weeks into the game it goes horribly.

05:22
💎 #
2

You can't just run an A/B test and look at D360 ARPU. Whenever you try to improve LTV you have to take a particular lens, a proxy for your success.

06:50
💎 #
3

It is very different to focus on maximizing LTV for users that have not yet installed the game vs. maximizing LTV for players that have been in the game for 6 months. Older players already have a system in their heads that they are used to, but with new users you can try a lot of "weird and wonderful" things.

08:10
💎 #
4

WildCard Games looks at D30 ARPU growth from install as north star metric. From experience, when focusing on earlier metrics they've seen great initial impact in the short term but that didn't play out in the long run.

08:36
💎 #
5

The downside of having D30 ARPU growth as a north star metric is that A/B tests require many weeks and tens of thousands of users. They currently have 6 or 7 A/B tests which means they're thinking about test ideas before even getting the results, which requires a lot of patience and discipline.

10:00
💎 #
6

You can't run new user tests on "older" players (6+ months), but the same principles of short/long term apply: increasing ARPDAU in the short term can end up hurting you. Example: if you do a great sale then you'll see great ARPDAU but there will be a hangover because players will have a lot of currency that they can't even go through fully.

11:42
💎 #
7

#7: 2 ways to deal with the fact that you can't know how A/B tests will play out in the long run:
1. "New user" testing **
2. **Testing things as part of an event
(LiveOps). Example: loyalty program where players would get a cat (in the game) with each purchase (they actually got users from the control group that wanted in).

12:20
💎 #
8

You do not want to alienate your most loyal users. Week-long events help players understand that things are one-off and subject to change. You can also target specific users (non-US, a particular segment, etc.). This allows you to do multiple iteration without harming users/numbers and also makes it easier to model (build the events inside your model). Example: test where you increase prices.

13:12
💎 #
9

The second you add a new feature to your game, all your models and previous A/B tests are no longer totally valid. It's the same thing with your ARPU curve.

16:16
💎 #
10

It's really important to have the product team set clear ARPU and LTV growth goals and communicate them to the UA team. Example: interstitial ad frequency test → the UA team needs to know that D1 ARPU might be great and not go 10x on spend.

17:22
💎 #
11

LTV curve is important, modeling is important. But the underrated thing is the communication between the UA and product team. The ARPU/LTV curves are backward-looking. Since the UA sources you're scaling can bring players with different behaviors (e.g. VO on Facebook vs. incentivized traffic or pre-installs) your product team needs to be aware of it to build the right features (e.g. focusing on whales features vs. ad features).

19:34
💎 #
12

You can have a game that doesn't monetize at all at the core. But assuming the core is really solid then if you have world-class sales and LiveOps you can monetize very well. From there you can take bets on how to improve your games with both features and LiveOps events.

22:07
💎 #
13

Before you can even start to focus on monetization, you have to make sure you have a very tight core game loop and economy. Otherwise it's like going on Amazon and trying to sell/optimize a product for which there is no demand.

22:45
💎 #
14

During Cat Games' soft launch they 10x'ed ARPDAU. Now they are actually doing A/B tests where they are clawing back some of the features they initially added in (because the economy of the game got out of control) and seeing great improvements.
_Example: took out global chat (it was also difficult to monitor) → early ARPDAU almost doubled and retention improved! _Watch to understand why.

23:52
💎 #
15

For decision-making on features, they've been using the concept of LTV vs. effort: LTV impact for days of efforts required. The same way the dev team defines how many days of efforts are necessary for a feature, the product team is responsible for creating the same thing for LTV impacts.

28:55
💎 #
16

They have exposed their north star metric of D30 ARPU growth to the team and set quarterly D30 ARPU growth goals at the game level, so the whole team is accountable. Benefits:
1. It gets buy-in on chasing LTV growth
2. Once you get better at predicting that, the UA team can think of how much they can spend now vs. what they will be able to spend.

30:06
💎 #
17

WildCard has started to look at finding a way to anti-test whatever they plan on testing. You can get great learnings: tests can still be wins AND you're learning about your audience (and your misconceptions). Example: decreasing ad load instead of increasing ad load.

32:55
💎 #
18

Most of the features introducing a source of coins ended up being negative in the long term, and it's even worse when the feature is fun.

36:37
💎 #
19

You need to run all these tests but you need to also actually talk to your players

39:07
💎 #
20

Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.

44:08
The gems from this resource are only available to premium members.
💎 #
1

The high level definition of LTV as ARPDAU*Retention has no application in the real world: changes in the game can have unforeseen side effects. You can improve ARPDAU but a few weeks into the game it goes horribly.

05:22
💎 #
2

You can't just run an A/B test and look at D360 ARPU. Whenever you try to improve LTV you have to take a particular lens, a proxy for your success.

06:50
💎 #
3

It is very different to focus on maximizing LTV for users that have not yet installed the game vs. maximizing LTV for players that have been in the game for 6 months. Older players already have a system in their heads that they are used to, but with new users you can try a lot of "weird and wonderful" things.

08:10
💎 #
4

WildCard Games looks at D30 ARPU growth from install as north star metric. From experience, when focusing on earlier metrics they've seen great initial impact in the short term but that didn't play out in the long run.

08:36
💎 #
5

The downside of having D30 ARPU growth as a north star metric is that A/B tests require many weeks and tens of thousands of users. They currently have 6 or 7 A/B tests which means they're thinking about test ideas before even getting the results, which requires a lot of patience and discipline.

10:00
💎 #
6

You can't run new user tests on "older" players (6+ months), but the same principles of short/long term apply: increasing ARPDAU in the short term can end up hurting you. Example: if you do a great sale then you'll see great ARPDAU but there will be a hangover because players will have a lot of currency that they can't even go through fully.

11:42
💎 #
7

#7: 2 ways to deal with the fact that you can't know how A/B tests will play out in the long run:
1. "New user" testing **
2. **Testing things as part of an event
(LiveOps). Example: loyalty program where players would get a cat (in the game) with each purchase (they actually got users from the control group that wanted in).

12:20
💎 #
8

You do not want to alienate your most loyal users. Week-long events help players understand that things are one-off and subject to change. You can also target specific users (non-US, a particular segment, etc.). This allows you to do multiple iteration without harming users/numbers and also makes it easier to model (build the events inside your model). Example: test where you increase prices.

13:12
💎 #
9

The second you add a new feature to your game, all your models and previous A/B tests are no longer totally valid. It's the same thing with your ARPU curve.

16:16
💎 #
10

It's really important to have the product team set clear ARPU and LTV growth goals and communicate them to the UA team. Example: interstitial ad frequency test → the UA team needs to know that D1 ARPU might be great and not go 10x on spend.

17:22
💎 #
11

LTV curve is important, modeling is important. But the underrated thing is the communication between the UA and product team. The ARPU/LTV curves are backward-looking. Since the UA sources you're scaling can bring players with different behaviors (e.g. VO on Facebook vs. incentivized traffic or pre-installs) your product team needs to be aware of it to build the right features (e.g. focusing on whales features vs. ad features).

19:34
💎 #
12

You can have a game that doesn't monetize at all at the core. But assuming the core is really solid then if you have world-class sales and LiveOps you can monetize very well. From there you can take bets on how to improve your games with both features and LiveOps events.

22:07
💎 #
13

Before you can even start to focus on monetization, you have to make sure you have a very tight core game loop and economy. Otherwise it's like going on Amazon and trying to sell/optimize a product for which there is no demand.

22:45
💎 #
14

During Cat Games' soft launch they 10x'ed ARPDAU. Now they are actually doing A/B tests where they are clawing back some of the features they initially added in (because the economy of the game got out of control) and seeing great improvements.
_Example: took out global chat (it was also difficult to monitor) → early ARPDAU almost doubled and retention improved! _Watch to understand why.

23:52
💎 #
15

For decision-making on features, they've been using the concept of LTV vs. effort: LTV impact for days of efforts required. The same way the dev team defines how many days of efforts are necessary for a feature, the product team is responsible for creating the same thing for LTV impacts.

28:55
💎 #
16

They have exposed their north star metric of D30 ARPU growth to the team and set quarterly D30 ARPU growth goals at the game level, so the whole team is accountable. Benefits:
1. It gets buy-in on chasing LTV growth
2. Once you get better at predicting that, the UA team can think of how much they can spend now vs. what they will be able to spend.

30:06
💎 #
17

WildCard has started to look at finding a way to anti-test whatever they plan on testing. You can get great learnings: tests can still be wins AND you're learning about your audience (and your misconceptions). Example: decreasing ad load instead of increasing ad load.

32:55
💎 #
18

Most of the features introducing a source of coins ended up being negative in the long term, and it's even worse when the feature is fun.

36:37
💎 #
19

You need to run all these tests but you need to also actually talk to your players

39:07
💎 #
20

Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.

44:08
The gems from this resource are only available to premium members.

Gems are the key bite-size insights "mined" from a specific mobile marketing resource, like a webinar, a panel or a podcast.
They allow you to save time by grasping the most important information in a couple of minutes, and also each include the timestamp from the source.

💎 #
1

The high level definition of LTV as ARPDAU*Retention has no application in the real world: changes in the game can have unforeseen side effects. You can improve ARPDAU but a few weeks into the game it goes horribly.

05:22
💎 #
2

You can't just run an A/B test and look at D360 ARPU. Whenever you try to improve LTV you have to take a particular lens, a proxy for your success.

06:50
💎 #
3

It is very different to focus on maximizing LTV for users that have not yet installed the game vs. maximizing LTV for players that have been in the game for 6 months. Older players already have a system in their heads that they are used to, but with new users you can try a lot of "weird and wonderful" things.

08:10
💎 #
4

WildCard Games looks at D30 ARPU growth from install as north star metric. From experience, when focusing on earlier metrics they've seen great initial impact in the short term but that didn't play out in the long run.

08:36
💎 #
5

The downside of having D30 ARPU growth as a north star metric is that A/B tests require many weeks and tens of thousands of users. They currently have 6 or 7 A/B tests which means they're thinking about test ideas before even getting the results, which requires a lot of patience and discipline.

10:00
💎 #
6

You can't run new user tests on "older" players (6+ months), but the same principles of short/long term apply: increasing ARPDAU in the short term can end up hurting you. Example: if you do a great sale then you'll see great ARPDAU but there will be a hangover because players will have a lot of currency that they can't even go through fully.

11:42
💎 #
7

#7: 2 ways to deal with the fact that you can't know how A/B tests will play out in the long run:
1. "New user" testing **
2. **Testing things as part of an event
(LiveOps). Example: loyalty program where players would get a cat (in the game) with each purchase (they actually got users from the control group that wanted in).

12:20
💎 #
8

You do not want to alienate your most loyal users. Week-long events help players understand that things are one-off and subject to change. You can also target specific users (non-US, a particular segment, etc.). This allows you to do multiple iteration without harming users/numbers and also makes it easier to model (build the events inside your model). Example: test where you increase prices.

13:12
💎 #
9

The second you add a new feature to your game, all your models and previous A/B tests are no longer totally valid. It's the same thing with your ARPU curve.

16:16
💎 #
10

It's really important to have the product team set clear ARPU and LTV growth goals and communicate them to the UA team. Example: interstitial ad frequency test → the UA team needs to know that D1 ARPU might be great and not go 10x on spend.

17:22
💎 #
11

LTV curve is important, modeling is important. But the underrated thing is the communication between the UA and product team. The ARPU/LTV curves are backward-looking. Since the UA sources you're scaling can bring players with different behaviors (e.g. VO on Facebook vs. incentivized traffic or pre-installs) your product team needs to be aware of it to build the right features (e.g. focusing on whales features vs. ad features).

19:34
💎 #
12

You can have a game that doesn't monetize at all at the core. But assuming the core is really solid then if you have world-class sales and LiveOps you can monetize very well. From there you can take bets on how to improve your games with both features and LiveOps events.

22:07
💎 #
13

Before you can even start to focus on monetization, you have to make sure you have a very tight core game loop and economy. Otherwise it's like going on Amazon and trying to sell/optimize a product for which there is no demand.

22:45
💎 #
14

During Cat Games' soft launch they 10x'ed ARPDAU. Now they are actually doing A/B tests where they are clawing back some of the features they initially added in (because the economy of the game got out of control) and seeing great improvements.
_Example: took out global chat (it was also difficult to monitor) → early ARPDAU almost doubled and retention improved! _Watch to understand why.

23:52
💎 #
15

For decision-making on features, they've been using the concept of LTV vs. effort: LTV impact for days of efforts required. The same way the dev team defines how many days of efforts are necessary for a feature, the product team is responsible for creating the same thing for LTV impacts.

28:55
💎 #
16

They have exposed their north star metric of D30 ARPU growth to the team and set quarterly D30 ARPU growth goals at the game level, so the whole team is accountable. Benefits:
1. It gets buy-in on chasing LTV growth
2. Once you get better at predicting that, the UA team can think of how much they can spend now vs. what they will be able to spend.

30:06
💎 #
17

WildCard has started to look at finding a way to anti-test whatever they plan on testing. You can get great learnings: tests can still be wins AND you're learning about your audience (and your misconceptions). Example: decreasing ad load instead of increasing ad load.

32:55
💎 #
18

Most of the features introducing a source of coins ended up being negative in the long term, and it's even worse when the feature is fun.

36:37
💎 #
19

You need to run all these tests but you need to also actually talk to your players

39:07
💎 #
20

Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.

44:08

Notes for this resource are currently being transferred and will be available soon.

Description for LTV


Josh

  • At the high level, LTV is ARPDAU*Retention
  • [💎 @05:22] The high level definition of LTV as ARPDAU*Retention has no application into the real world: changes in the game can have unforeseen side effects. You can improve ARPDAU but a few weeks into the game it goes horribly.
  • I like to think about LTV as this big beautiful ARPU curve that floats up in the horizon forever


Sasha

  • You have a nice model but it doesn't pan out at all. Games are complicated system with tons of levers.
  • [💎 @06:50] You can't just run an A/B test and look at D360 ARPU. Whenever you try to improve LTV you have to take a particular lense, a proxy for your success.
  • What lense? ARPDAU can be very deceptive: it can start looking great, but it might kill retention.


Lense/proxy you take when trying to measure the changes you make to your ARPU?


Josh

  • Believe that most of their lifetime userbase has not yet installed the game so a lot of focus is on maximizing LTV for new users. Which is different than maximizing LTV for players that have been in the game for 6 months.
  • [💎 @08:10] It is very different to focus on maximizing LTV for users that have not yet installed the game than on maximizing LTV for players that have been in the game for 6 months. Older players already have a system in their heads that they are used to, but with new users you can try a lot of new "weird and wonderful" things.
  • [💎 @08:36] WildCard Games looks at D30 ARPU growth from install. From experience, when focusing on earlier metrics they've seen great initial impact in the short term but that didn't play out in the long run.
  • Detailed reasoning: the best way he's found to maximize ARPDAU is to nuke the LTV: show ads every couple of seconds (if ad based) or some really good offers early in the game (if IAP monetized). The problem is that when you start looking at 6-12 months, how does that affect metrics? They've tried changes to the game that were very positive for D1/D7 but starting D30 they see players getting tired.
  • [💎 @10:00] The downside of having D30 ARPU growth as a north star metric is that A/B tests require many weeks and tens of thousands of users. They currently have 6 or 7 A/B tests which means they're thinking about test ideas before even getting the results, which requires a lot of patience and discipline.

You're filling up this bus with users to see how far the bus goes


How to model out A/B tests when you have no idea how it will play out on D365?


Sasha

  • Testing new users is one way way to A/B test
  • Most of their games have been around for more than 6 months, and these "older players" is who they get the most revenue from.
  • [💎 @11:42] You can't run new user tests on "older" players (6+ months), but the same principles of short/long term apply: increasing ARPDAU in the short term can end up hurting you. Example: if you do a great sale then you'll see great ARPDAU but there will be a hangover because players will have a lot of currency that they can't even use.
  • You have to not only test on a long horizon and measure for a long period of time, you also have look at all of the metrics.
  • [💎 @12:20] 2 ways to deal with the fact that you can't know how A/B tests will play out in the long run:
  1. "New user" testing
  2. Test things as part of an event (LiveOps). Example: loyalty program where players would get a cat (in the game) with each purchase (they actually got users from the control group that wanted in).
  • If instead of A/B testing new things you directly roll out offers or pricing tests, you don't get to iterate (which is key to make sure your test works).
  • [💎 @13:12] You do not want to alienate your most loyal users. Week-long events help players understand that things are one-off and subject to change. You can also target specific users (non-US, a particular segment, etc.). This allows you to do multiple iteration without harming users/numbers and also makes it easier to model (build the events inside your model). Example: test where you increase prices.


Josh

  • [💎 @16:16] The second you add a new feature to your game, all your models and previous A/B tests are no longer totally valid. It's the same thing with your ARPU curve.
  • The product team takes care of new users the UA team brings them.
  • [💎 @17:22] It's really important to have the product team set clear ARPU and LTV growth goals and communicate that to the UA team. Example: interstitial ad frequency test → the UA team needs to know that D1 ARPU might be great and not go 10x on spend.
  • They had a product placement from Mistplay which brought a great source of new users, but also really different. Players come in and spend straight away and payback time is much faster, however having 3x D1/D7 ARPU on that source doesn't mean that the LTV will be.
  • [💎 @19:34] LTV curve is important, modeling is important. But the underrated thing is the communication between the UA and product team. However the ARPU/LTV curves are backward-looking Since the UA sources you're scaling can bring players with different behaviors (e.g. VO on Facebook vs. incentivized traffic or pre-installs) your product team needs to be aware of it to build the right features (e.g. focusing on whales features vs. ad features).


Monetization first or features first?


Sasha

  • Sasha has changed his mind on this. Before: build a game that monetizes very well and where player will spend a bunch on. But not really true.
  • The core mechanics/loop may have almost 0 monetization (besides speed ups). Now there are a lot of games with no friction points and instead have 2 layers: sales and liveops.
  • [💎 @22:07] You can have a game that doesn't monetize at all at the core. But assuming the core is really solid then if you have world-class sales and LiveOps you can monetize very well. From there you can take bets on how to improve your games with both features and LiveOps events.
  • [💎 @22:45] Before you can even start to focus on monetization, you have to make sure you have a very tight core game loop and economy. Otherwise it's like going on Amazon and trying to sell/optimize a product for which there is no demand.
  • If players get a lot of coins in your game but can't go through all of them, they'll never buy coins.
  • [💎 @23:52] During Cat Games' soft launch they 10X'ed ARPDAU. Now they are actually doing A/B tests where they are clawing back some of the features they initially added in (because the economy got out of control) and seeing great improvements. Example: took out global chat (it was difficult to monitor). Early ARPDAU almost doubled and retention improved! Watch to understand why.


Josh

  • Launched a weekend tournament feature. They usually always wait to D30 but this feature was so great (players spending much more than they sourced) that they rolled it out after two weekends. Unfortunately it had unforeseen economy changes: after D20 WildCard realized that they were now sourcing more than they were spending. Now they have to find creative ways to fix that because they did not A/B test.
  • Chasing big flashy features is something to be careful of.
  • [💎 @28:55] For decision-making on features, they've been using the concept of LTV vs. effort: LTV impact for days of efforts. The same way the dev team defines how many days of efforts are necessary for a feature, the product team is responsible for creating the same thing for LTV impacts.
  • It is not perfect (LTV can't always be trusted) but the idea is to still improve decision-making overall.
  • [💎 @30:06] They have exposed their north star metric of D30 ARPU growth to the team and set quarterly D30 ARPU growth goals at the game level, so the whole team is accountable.
  • 1. It gets buy-in on chasing LTV growth
  • 2. Once you get better at predicting that, the UA team can think of how much they can spend now vs. what they will be able to spend.


Tests they thought would go great and failed?


It happens a lot!


Josh

  • [💎 @32:55] WildCard has started to look at finding a way to anti-test to whatever they plan on testing. You can get great learnings: they can still be wins AND you're learning about your audience. Example: decreasing ad load instead of increasing ad load.
  • They once tried adding a piggy bank feature with a flow inspired by the top 5 grossing games. It reduced LTV drastically. After feedback from players they managed to improve it a lot but it still failed overall and it took a lot of effort.


Sasha

  • Early on they were shipping a lot of things. Some of them they thought were great but backfired.
  • [💎 @36:37] Most of the features introducing a source of coins ended up being negative in the long term, and it's even worse when the feature is fun.
  • Making the game more easy doesn't always get more people to harder levels. They've actually seen the opposite.
  • Tested a loyalty program (you buy something, you get something) after sitting down with 20 of their best players and going through all of their ideas.
  • [💎 @39:07] You need to run all these tests but you need to also actually talk to your players.


Almost every time I've done something great it's been preceded by having a meaningful conversation with some of our top users.


LiveOps are important in gaming. What can other apps learn from that?


Sasha

  • Free 2 play is like building an ecommerce site: same tactics, etc. What they run are the same stuff as in retail: sales, loyalty events, etc. It applies both
  • Games have been meaningfully informed by some of the best sales and LiveOps tactics from both old school and modern retail.
  • Core principles of that:
  • People love novelty. What would have happened if Pokemon were not adding new Pokemon? You need to always add new content.
  • Examples: Mino Games introduces cats every week and people are very excited. Zara shortens the cycle at which they introduce new products.
  • Sales tactics used across the board: piggy bank, loyalty programs, etc.
  • Aspect of community which is very important for LiveOps. People love to have someone to do this with.
  • [💎 @44:08] Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.


Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.


Josh

  • Novelty is absolutely critical. One of the most valuable ways to use that novelty is to build habit and loyalty and finding ways to make sure that is durable. Example: there is a new cat every week.
  • E-commerce example: Amazon daily deals was a great re-engagement hook.
  • It's tempting to chase the topline but stacking lots of little things that are incrementally re-usable (or cheap) can often be very valuable as well. Look at the opportunity cost when considering new features or creative assets: re-usable things will save up time in the future.


The notes from this resource are only available to premium members.

Description for LTV


Josh

  • At the high level, LTV is ARPDAU*Retention
  • [💎 @05:22] The high level definition of LTV as ARPDAU*Retention has no application into the real world: changes in the game can have unforeseen side effects. You can improve ARPDAU but a few weeks into the game it goes horribly.
  • I like to think about LTV as this big beautiful ARPU curve that floats up in the horizon forever


Sasha

  • You have a nice model but it doesn't pan out at all. Games are complicated system with tons of levers.
  • [💎 @06:50] You can't just run an A/B test and look at D360 ARPU. Whenever you try to improve LTV you have to take a particular lense, a proxy for your success.
  • What lense? ARPDAU can be very deceptive: it can start looking great, but it might kill retention.


Lense/proxy you take when trying to measure the changes you make to your ARPU?


Josh

  • Believe that most of their lifetime userbase has not yet installed the game so a lot of focus is on maximizing LTV for new users. Which is different than maximizing LTV for players that have been in the game for 6 months.
  • [💎 @08:10] It is very different to focus on maximizing LTV for users that have not yet installed the game than on maximizing LTV for players that have been in the game for 6 months. Older players already have a system in their heads that they are used to, but with new users you can try a lot of new "weird and wonderful" things.
  • [💎 @08:36] WildCard Games looks at D30 ARPU growth from install. From experience, when focusing on earlier metrics they've seen great initial impact in the short term but that didn't play out in the long run.
  • Detailed reasoning: the best way he's found to maximize ARPDAU is to nuke the LTV: show ads every couple of seconds (if ad based) or some really good offers early in the game (if IAP monetized). The problem is that when you start looking at 6-12 months, how does that affect metrics? They've tried changes to the game that were very positive for D1/D7 but starting D30 they see players getting tired.
  • [💎 @10:00] The downside of having D30 ARPU growth as a north star metric is that A/B tests require many weeks and tens of thousands of users. They currently have 6 or 7 A/B tests which means they're thinking about test ideas before even getting the results, which requires a lot of patience and discipline.

You're filling up this bus with users to see how far the bus goes


How to model out A/B tests when you have no idea how it will play out on D365?


Sasha

  • Testing new users is one way way to A/B test
  • Most of their games have been around for more than 6 months, and these "older players" is who they get the most revenue from.
  • [💎 @11:42] You can't run new user tests on "older" players (6+ months), but the same principles of short/long term apply: increasing ARPDAU in the short term can end up hurting you. Example: if you do a great sale then you'll see great ARPDAU but there will be a hangover because players will have a lot of currency that they can't even use.
  • You have to not only test on a long horizon and measure for a long period of time, you also have look at all of the metrics.
  • [💎 @12:20] 2 ways to deal with the fact that you can't know how A/B tests will play out in the long run:
  1. "New user" testing
  2. Test things as part of an event (LiveOps). Example: loyalty program where players would get a cat (in the game) with each purchase (they actually got users from the control group that wanted in).
  • If instead of A/B testing new things you directly roll out offers or pricing tests, you don't get to iterate (which is key to make sure your test works).
  • [💎 @13:12] You do not want to alienate your most loyal users. Week-long events help players understand that things are one-off and subject to change. You can also target specific users (non-US, a particular segment, etc.). This allows you to do multiple iteration without harming users/numbers and also makes it easier to model (build the events inside your model). Example: test where you increase prices.


Josh

  • [💎 @16:16] The second you add a new feature to your game, all your models and previous A/B tests are no longer totally valid. It's the same thing with your ARPU curve.
  • The product team takes care of new users the UA team brings them.
  • [💎 @17:22] It's really important to have the product team set clear ARPU and LTV growth goals and communicate that to the UA team. Example: interstitial ad frequency test → the UA team needs to know that D1 ARPU might be great and not go 10x on spend.
  • They had a product placement from Mistplay which brought a great source of new users, but also really different. Players come in and spend straight away and payback time is much faster, however having 3x D1/D7 ARPU on that source doesn't mean that the LTV will be.
  • [💎 @19:34] LTV curve is important, modeling is important. But the underrated thing is the communication between the UA and product team. However the ARPU/LTV curves are backward-looking Since the UA sources you're scaling can bring players with different behaviors (e.g. VO on Facebook vs. incentivized traffic or pre-installs) your product team needs to be aware of it to build the right features (e.g. focusing on whales features vs. ad features).


Monetization first or features first?


Sasha

  • Sasha has changed his mind on this. Before: build a game that monetizes very well and where player will spend a bunch on. But not really true.
  • The core mechanics/loop may have almost 0 monetization (besides speed ups). Now there are a lot of games with no friction points and instead have 2 layers: sales and liveops.
  • [💎 @22:07] You can have a game that doesn't monetize at all at the core. But assuming the core is really solid then if you have world-class sales and LiveOps you can monetize very well. From there you can take bets on how to improve your games with both features and LiveOps events.
  • [💎 @22:45] Before you can even start to focus on monetization, you have to make sure you have a very tight core game loop and economy. Otherwise it's like going on Amazon and trying to sell/optimize a product for which there is no demand.
  • If players get a lot of coins in your game but can't go through all of them, they'll never buy coins.
  • [💎 @23:52] During Cat Games' soft launch they 10X'ed ARPDAU. Now they are actually doing A/B tests where they are clawing back some of the features they initially added in (because the economy got out of control) and seeing great improvements. Example: took out global chat (it was difficult to monitor). Early ARPDAU almost doubled and retention improved! Watch to understand why.


Josh

  • Launched a weekend tournament feature. They usually always wait to D30 but this feature was so great (players spending much more than they sourced) that they rolled it out after two weekends. Unfortunately it had unforeseen economy changes: after D20 WildCard realized that they were now sourcing more than they were spending. Now they have to find creative ways to fix that because they did not A/B test.
  • Chasing big flashy features is something to be careful of.
  • [💎 @28:55] For decision-making on features, they've been using the concept of LTV vs. effort: LTV impact for days of efforts. The same way the dev team defines how many days of efforts are necessary for a feature, the product team is responsible for creating the same thing for LTV impacts.
  • It is not perfect (LTV can't always be trusted) but the idea is to still improve decision-making overall.
  • [💎 @30:06] They have exposed their north star metric of D30 ARPU growth to the team and set quarterly D30 ARPU growth goals at the game level, so the whole team is accountable.
  • 1. It gets buy-in on chasing LTV growth
  • 2. Once you get better at predicting that, the UA team can think of how much they can spend now vs. what they will be able to spend.


Tests they thought would go great and failed?


It happens a lot!


Josh

  • [💎 @32:55] WildCard has started to look at finding a way to anti-test to whatever they plan on testing. You can get great learnings: they can still be wins AND you're learning about your audience. Example: decreasing ad load instead of increasing ad load.
  • They once tried adding a piggy bank feature with a flow inspired by the top 5 grossing games. It reduced LTV drastically. After feedback from players they managed to improve it a lot but it still failed overall and it took a lot of effort.


Sasha

  • Early on they were shipping a lot of things. Some of them they thought were great but backfired.
  • [💎 @36:37] Most of the features introducing a source of coins ended up being negative in the long term, and it's even worse when the feature is fun.
  • Making the game more easy doesn't always get more people to harder levels. They've actually seen the opposite.
  • Tested a loyalty program (you buy something, you get something) after sitting down with 20 of their best players and going through all of their ideas.
  • [💎 @39:07] You need to run all these tests but you need to also actually talk to your players.


Almost every time I've done something great it's been preceded by having a meaningful conversation with some of our top users.


LiveOps are important in gaming. What can other apps learn from that?


Sasha

  • Free 2 play is like building an ecommerce site: same tactics, etc. What they run are the same stuff as in retail: sales, loyalty events, etc. It applies both
  • Games have been meaningfully informed by some of the best sales and LiveOps tactics from both old school and modern retail.
  • Core principles of that:
  • People love novelty. What would have happened if Pokemon were not adding new Pokemon? You need to always add new content.
  • Examples: Mino Games introduces cats every week and people are very excited. Zara shortens the cycle at which they introduce new products.
  • Sales tactics used across the board: piggy bank, loyalty programs, etc.
  • Aspect of community which is very important for LiveOps. People love to have someone to do this with.
  • [💎 @44:08] Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.


Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.


Josh

  • Novelty is absolutely critical. One of the most valuable ways to use that novelty is to build habit and loyalty and finding ways to make sure that is durable. Example: there is a new cat every week.
  • E-commerce example: Amazon daily deals was a great re-engagement hook.
  • It's tempting to chase the topline but stacking lots of little things that are incrementally re-usable (or cheap) can often be very valuable as well. Look at the opportunity cost when considering new features or creative assets: re-usable things will save up time in the future.


The notes from this resource are only available to premium members.

Description for LTV


Josh

  • At the high level, LTV is ARPDAU*Retention
  • [💎 @05:22] The high level definition of LTV as ARPDAU*Retention has no application into the real world: changes in the game can have unforeseen side effects. You can improve ARPDAU but a few weeks into the game it goes horribly.
  • I like to think about LTV as this big beautiful ARPU curve that floats up in the horizon forever


Sasha

  • You have a nice model but it doesn't pan out at all. Games are complicated system with tons of levers.
  • [💎 @06:50] You can't just run an A/B test and look at D360 ARPU. Whenever you try to improve LTV you have to take a particular lense, a proxy for your success.
  • What lense? ARPDAU can be very deceptive: it can start looking great, but it might kill retention.


Lense/proxy you take when trying to measure the changes you make to your ARPU?


Josh

  • Believe that most of their lifetime userbase has not yet installed the game so a lot of focus is on maximizing LTV for new users. Which is different than maximizing LTV for players that have been in the game for 6 months.
  • [💎 @08:10] It is very different to focus on maximizing LTV for users that have not yet installed the game than on maximizing LTV for players that have been in the game for 6 months. Older players already have a system in their heads that they are used to, but with new users you can try a lot of new "weird and wonderful" things.
  • [💎 @08:36] WildCard Games looks at D30 ARPU growth from install. From experience, when focusing on earlier metrics they've seen great initial impact in the short term but that didn't play out in the long run.
  • Detailed reasoning: the best way he's found to maximize ARPDAU is to nuke the LTV: show ads every couple of seconds (if ad based) or some really good offers early in the game (if IAP monetized). The problem is that when you start looking at 6-12 months, how does that affect metrics? They've tried changes to the game that were very positive for D1/D7 but starting D30 they see players getting tired.
  • [💎 @10:00] The downside of having D30 ARPU growth as a north star metric is that A/B tests require many weeks and tens of thousands of users. They currently have 6 or 7 A/B tests which means they're thinking about test ideas before even getting the results, which requires a lot of patience and discipline.

You're filling up this bus with users to see how far the bus goes


How to model out A/B tests when you have no idea how it will play out on D365?


Sasha

  • Testing new users is one way way to A/B test
  • Most of their games have been around for more than 6 months, and these "older players" is who they get the most revenue from.
  • [💎 @11:42] You can't run new user tests on "older" players (6+ months), but the same principles of short/long term apply: increasing ARPDAU in the short term can end up hurting you. Example: if you do a great sale then you'll see great ARPDAU but there will be a hangover because players will have a lot of currency that they can't even use.
  • You have to not only test on a long horizon and measure for a long period of time, you also have look at all of the metrics.
  • [💎 @12:20] 2 ways to deal with the fact that you can't know how A/B tests will play out in the long run:
  1. "New user" testing
  2. Test things as part of an event (LiveOps). Example: loyalty program where players would get a cat (in the game) with each purchase (they actually got users from the control group that wanted in).
  • If instead of A/B testing new things you directly roll out offers or pricing tests, you don't get to iterate (which is key to make sure your test works).
  • [💎 @13:12] You do not want to alienate your most loyal users. Week-long events help players understand that things are one-off and subject to change. You can also target specific users (non-US, a particular segment, etc.). This allows you to do multiple iteration without harming users/numbers and also makes it easier to model (build the events inside your model). Example: test where you increase prices.


Josh

  • [💎 @16:16] The second you add a new feature to your game, all your models and previous A/B tests are no longer totally valid. It's the same thing with your ARPU curve.
  • The product team takes care of new users the UA team brings them.
  • [💎 @17:22] It's really important to have the product team set clear ARPU and LTV growth goals and communicate that to the UA team. Example: interstitial ad frequency test → the UA team needs to know that D1 ARPU might be great and not go 10x on spend.
  • They had a product placement from Mistplay which brought a great source of new users, but also really different. Players come in and spend straight away and payback time is much faster, however having 3x D1/D7 ARPU on that source doesn't mean that the LTV will be.
  • [💎 @19:34] LTV curve is important, modeling is important. But the underrated thing is the communication between the UA and product team. However the ARPU/LTV curves are backward-looking Since the UA sources you're scaling can bring players with different behaviors (e.g. VO on Facebook vs. incentivized traffic or pre-installs) your product team needs to be aware of it to build the right features (e.g. focusing on whales features vs. ad features).


Monetization first or features first?


Sasha

  • Sasha has changed his mind on this. Before: build a game that monetizes very well and where player will spend a bunch on. But not really true.
  • The core mechanics/loop may have almost 0 monetization (besides speed ups). Now there are a lot of games with no friction points and instead have 2 layers: sales and liveops.
  • [💎 @22:07] You can have a game that doesn't monetize at all at the core. But assuming the core is really solid then if you have world-class sales and LiveOps you can monetize very well. From there you can take bets on how to improve your games with both features and LiveOps events.
  • [💎 @22:45] Before you can even start to focus on monetization, you have to make sure you have a very tight core game loop and economy. Otherwise it's like going on Amazon and trying to sell/optimize a product for which there is no demand.
  • If players get a lot of coins in your game but can't go through all of them, they'll never buy coins.
  • [💎 @23:52] During Cat Games' soft launch they 10X'ed ARPDAU. Now they are actually doing A/B tests where they are clawing back some of the features they initially added in (because the economy got out of control) and seeing great improvements. Example: took out global chat (it was difficult to monitor). Early ARPDAU almost doubled and retention improved! Watch to understand why.


Josh

  • Launched a weekend tournament feature. They usually always wait to D30 but this feature was so great (players spending much more than they sourced) that they rolled it out after two weekends. Unfortunately it had unforeseen economy changes: after D20 WildCard realized that they were now sourcing more than they were spending. Now they have to find creative ways to fix that because they did not A/B test.
  • Chasing big flashy features is something to be careful of.
  • [💎 @28:55] For decision-making on features, they've been using the concept of LTV vs. effort: LTV impact for days of efforts. The same way the dev team defines how many days of efforts are necessary for a feature, the product team is responsible for creating the same thing for LTV impacts.
  • It is not perfect (LTV can't always be trusted) but the idea is to still improve decision-making overall.
  • [💎 @30:06] They have exposed their north star metric of D30 ARPU growth to the team and set quarterly D30 ARPU growth goals at the game level, so the whole team is accountable.
  • 1. It gets buy-in on chasing LTV growth
  • 2. Once you get better at predicting that, the UA team can think of how much they can spend now vs. what they will be able to spend.


Tests they thought would go great and failed?


It happens a lot!


Josh

  • [💎 @32:55] WildCard has started to look at finding a way to anti-test to whatever they plan on testing. You can get great learnings: they can still be wins AND you're learning about your audience. Example: decreasing ad load instead of increasing ad load.
  • They once tried adding a piggy bank feature with a flow inspired by the top 5 grossing games. It reduced LTV drastically. After feedback from players they managed to improve it a lot but it still failed overall and it took a lot of effort.


Sasha

  • Early on they were shipping a lot of things. Some of them they thought were great but backfired.
  • [💎 @36:37] Most of the features introducing a source of coins ended up being negative in the long term, and it's even worse when the feature is fun.
  • Making the game more easy doesn't always get more people to harder levels. They've actually seen the opposite.
  • Tested a loyalty program (you buy something, you get something) after sitting down with 20 of their best players and going through all of their ideas.
  • [💎 @39:07] You need to run all these tests but you need to also actually talk to your players.


Almost every time I've done something great it's been preceded by having a meaningful conversation with some of our top users.


LiveOps are important in gaming. What can other apps learn from that?


Sasha

  • Free 2 play is like building an ecommerce site: same tactics, etc. What they run are the same stuff as in retail: sales, loyalty events, etc. It applies both
  • Games have been meaningfully informed by some of the best sales and LiveOps tactics from both old school and modern retail.
  • Core principles of that:
  • People love novelty. What would have happened if Pokemon were not adding new Pokemon? You need to always add new content.
  • Examples: Mino Games introduces cats every week and people are very excited. Zara shortens the cycle at which they introduce new products.
  • Sales tactics used across the board: piggy bank, loyalty programs, etc.
  • Aspect of community which is very important for LiveOps. People love to have someone to do this with.
  • [💎 @44:08] Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.


Almost any product can learn from how games run LiveOps, the same way gaming learned from retail.


Josh

  • Novelty is absolutely critical. One of the most valuable ways to use that novelty is to build habit and loyalty and finding ways to make sure that is durable. Example: there is a new cat every week.
  • E-commerce example: Amazon daily deals was a great re-engagement hook.
  • It's tempting to chase the topline but stacking lots of little things that are incrementally re-usable (or cheap) can often be very valuable as well. Look at the opportunity cost when considering new features or creative assets: re-usable things will save up time in the future.