Understanding, Optimizing and Predicting LTV in Mobile Gaming

Item not bookmarked
Resource bookmarked
Bookmarking...
⛏️
Guest Miner:
Sylvain Gauchet
Review star
Review star
Review star
Review star
Review star
💎  x
16

Robert Magyar (Data Science Lead at SuperScale - Mobile Games) discusses how LTV predictions can help you and how to optimize LiveOps/Offers and Ivan Kozyev (Head of Analytics at Crazy Panda - Mobile Games) explains how to develop an effective LTV model for each stage of your game.

Source:
Understanding, Optimizing and Predicting LTV in Mobile Gaming
(no direct link to watch/listen)
(direct link to watch/listen)
Type:
Webinar
Publication date:
August 12, 2020
Added to the Vault on:
September 8, 2020
Invite a guest
These insights were shared through the free Growth Gems newsletter.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
💎 #
1

Chart your creatives on a X axis and check your D3/D7/D28 ROAS to quickly spot outlier creatives (both good and bad) so you can act on that (by reallocating spend for example). Example chart here.

10:15
💎 #
2

A benchmark comparing ROAS (e.g. D7/D28) for each week (X axis) with success thresholds allows you to evaluate your UA strategies. Example chart here.

11:29
💎 #
3

Understanding the CPI to spend relationship is a key factor in understanding UA payback and how you can scale your campaign. Example chart here.

13:54
💎 #
4

Questions you need to ask yourself to find the most profitable players (for LAL on Android for example - changes coming to iOS):
1. Is this group great at buying IAPs, do they do it frequently?
2. Is this group heavily engaged, does their engagement grow over time?
3. Do this group of players watch ads frequently? Make sure you have a benchmark to compare these new LAL/audiences to.

19:13
💎 #
5

Do not create too many groups/segments of players when looking at LTV. You need to make sure they are different so you can understand which group is better. It is not enough to segment based on how much they purchase however, you need to use other attributes too.

21:45
💎 #
6

If your LTV curve looks like a step function with jumps, either your game is relying mainly on LiveOps offers (not ideal design) or the amount of payers/players is too low. Example chart here.

24:12
💎 #
7

Your special offers are great if you can increase revenue per user while minimizing the discount. The value is in the personalization and showing the right offer (with a relevant content and price) at the right time. Example of special offer delivery system.

30:50
💎 #
8

Predicting LTV is different at different game stages: soft launch, some time after global launch or when the game is at maturity. See characteristics and suggested approach here.

42:58
💎 #
9

In soft launch we do not have the whole LTV curve but we somehow need to calculate the lifetime length so product knowledge is crucial because you're extrapolating. You have to know:
- Monetization limits (depth)
- User behavior

52:00
💎 #
10

The most important step in LTV model development is your LTV model and forecast validation. Always have a validation sample to test the model against, and it also must be representative. Make sure you do not build the model to work especially well against your validation sample (i.e. "overfit").

54:39
💎 #
11

Some time after global launch you need different LTV models: country groups (Tier 1 vs. Tier 2 vs. Tier 3), acquisition sources and optimization types (Google Ads vs. video networks) and monetization types (in-apps vs. ad-based, or live ops vs. regular purchases).

58:10
💎 #
12

Always think about how the LTV model will be used. Example: LTV model for the UA team needs to be working with a small sample size so decisions can be made at the campaign level vs. LTV model for strategic decisions needs to be more accurate and can be more thorough.

01:07:02
💎 #
13

If you are encountering issues when leveraging machine learning, build a quick model with rough "soft launch techniques" for quick validation. Have a few models using very limited amount of data so you can retrain the machine learning models as soon as possible.

01:10:18
💎 #
14

Understanding the impact of LiveOps events on LTV is difficult when only 3 or 4 LiveOps have been done. You can avoid having to wait by looking at peaks during the LiveOps event. "Slice" the LTV curve into smaller periods, define a validation cohort for each slice and calculate the impact on LTV over the period. Then normalize the impact and calculate the final improvement. Example chart here

01:12:21
💎 #
15

When evaluating the impact of LiveOps on LTV, do not forget to take into account the novelty effect: peaks tend to be higher during the first LiveOps events.

01:13:45
💎 #
16

For special offers LTV prediction models start with a rule-based system, then a probabilistic system and finally machine learning system. With the rule-based system it is less risky and more transparent which also helps you identify the impact of the changes you make.

01:21:55
The "gems" from this resource are only available to premium members.
  • Unlock access to gems from over 145 mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the members-only community.
Upgrade Your Plan
💎 #
1

Chart your creatives on a X axis and check your D3/D7/D28 ROAS to quickly spot outlier creatives (both good and bad) so you can act on that (by reallocating spend for example). Example chart here.

10:15
💎 #
2

A benchmark comparing ROAS (e.g. D7/D28) for each week (X axis) with success thresholds allows you to evaluate your UA strategies. Example chart here.

11:29
💎 #
3

Understanding the CPI to spend relationship is a key factor in understanding UA payback and how you can scale your campaign. Example chart here.

13:54
💎 #
4

Questions you need to ask yourself to find the most profitable players (for LAL on Android for example - changes coming to iOS):
1. Is this group great at buying IAPs, do they do it frequently?
2. Is this group heavily engaged, does their engagement grow over time?
3. Do this group of players watch ads frequently? Make sure you have a benchmark to compare these new LAL/audiences to.

19:13
💎 #
5

Do not create too many groups/segments of players when looking at LTV. You need to make sure they are different so you can understand which group is better. It is not enough to segment based on how much they purchase however, you need to use other attributes too.

21:45
💎 #
6

If your LTV curve looks like a step function with jumps, either your game is relying mainly on LiveOps offers (not ideal design) or the amount of payers/players is too low. Example chart here.

24:12
💎 #
7

Your special offers are great if you can increase revenue per user while minimizing the discount. The value is in the personalization and showing the right offer (with a relevant content and price) at the right time. Example of special offer delivery system.

30:50
💎 #
8

Predicting LTV is different at different game stages: soft launch, some time after global launch or when the game is at maturity. See characteristics and suggested approach here.

42:58
💎 #
9

In soft launch we do not have the whole LTV curve but we somehow need to calculate the lifetime length so product knowledge is crucial because you're extrapolating. You have to know:
- Monetization limits (depth)
- User behavior

52:00
💎 #
10

The most important step in LTV model development is your LTV model and forecast validation. Always have a validation sample to test the model against, and it also must be representative. Make sure you do not build the model to work especially well against your validation sample (i.e. "overfit").

54:39
💎 #
11

Some time after global launch you need different LTV models: country groups (Tier 1 vs. Tier 2 vs. Tier 3), acquisition sources and optimization types (Google Ads vs. video networks) and monetization types (in-apps vs. ad-based, or live ops vs. regular purchases).

58:10
💎 #
12

Always think about how the LTV model will be used. Example: LTV model for the UA team needs to be working with a small sample size so decisions can be made at the campaign level vs. LTV model for strategic decisions needs to be more accurate and can be more thorough.

01:07:02
💎 #
13

If you are encountering issues when leveraging machine learning, build a quick model with rough "soft launch techniques" for quick validation. Have a few models using very limited amount of data so you can retrain the machine learning models as soon as possible.

01:10:18
💎 #
14

Understanding the impact of LiveOps events on LTV is difficult when only 3 or 4 LiveOps have been done. You can avoid having to wait by looking at peaks during the LiveOps event. "Slice" the LTV curve into smaller periods, define a validation cohort for each slice and calculate the impact on LTV over the period. Then normalize the impact and calculate the final improvement. Example chart here

01:12:21
💎 #
15

When evaluating the impact of LiveOps on LTV, do not forget to take into account the novelty effect: peaks tend to be higher during the first LiveOps events.

01:13:45
💎 #
16

For special offers LTV prediction models start with a rule-based system, then a probabilistic system and finally machine learning system. With the rule-based system it is less risky and more transparent which also helps you identify the impact of the changes you make.

01:21:55
The "gems" from this resource are only available to premium members.

Gems are the key bite-size insights "mined" from a specific mobile marketing resource, like a webinar, a panel or a podcast.
They allow you to save time by grasping the most important information in a couple of minutes, and also each include the timestamp from the source.

Become a member to:
  • Unlock access to gems from over 145 mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the member-only community.
Request Access
💎 #
1

Chart your creatives on a X axis and check your D3/D7/D28 ROAS to quickly spot outlier creatives (both good and bad) so you can act on that (by reallocating spend for example). Example chart here.

10:15
💎 #
2

A benchmark comparing ROAS (e.g. D7/D28) for each week (X axis) with success thresholds allows you to evaluate your UA strategies. Example chart here.

11:29
💎 #
3

Understanding the CPI to spend relationship is a key factor in understanding UA payback and how you can scale your campaign. Example chart here.

13:54
💎 #
4

Questions you need to ask yourself to find the most profitable players (for LAL on Android for example - changes coming to iOS):
1. Is this group great at buying IAPs, do they do it frequently?
2. Is this group heavily engaged, does their engagement grow over time?
3. Do this group of players watch ads frequently? Make sure you have a benchmark to compare these new LAL/audiences to.

19:13
💎 #
5

Do not create too many groups/segments of players when looking at LTV. You need to make sure they are different so you can understand which group is better. It is not enough to segment based on how much they purchase however, you need to use other attributes too.

21:45
💎 #
6

If your LTV curve looks like a step function with jumps, either your game is relying mainly on LiveOps offers (not ideal design) or the amount of payers/players is too low. Example chart here.

24:12
💎 #
7

Your special offers are great if you can increase revenue per user while minimizing the discount. The value is in the personalization and showing the right offer (with a relevant content and price) at the right time. Example of special offer delivery system.

30:50
💎 #
8

Predicting LTV is different at different game stages: soft launch, some time after global launch or when the game is at maturity. See characteristics and suggested approach here.

42:58
💎 #
9

In soft launch we do not have the whole LTV curve but we somehow need to calculate the lifetime length so product knowledge is crucial because you're extrapolating. You have to know:
- Monetization limits (depth)
- User behavior

52:00
💎 #
10

The most important step in LTV model development is your LTV model and forecast validation. Always have a validation sample to test the model against, and it also must be representative. Make sure you do not build the model to work especially well against your validation sample (i.e. "overfit").

54:39
💎 #
11

Some time after global launch you need different LTV models: country groups (Tier 1 vs. Tier 2 vs. Tier 3), acquisition sources and optimization types (Google Ads vs. video networks) and monetization types (in-apps vs. ad-based, or live ops vs. regular purchases).

58:10
💎 #
12

Always think about how the LTV model will be used. Example: LTV model for the UA team needs to be working with a small sample size so decisions can be made at the campaign level vs. LTV model for strategic decisions needs to be more accurate and can be more thorough.

01:07:02
💎 #
13

If you are encountering issues when leveraging machine learning, build a quick model with rough "soft launch techniques" for quick validation. Have a few models using very limited amount of data so you can retrain the machine learning models as soon as possible.

01:10:18
💎 #
14

Understanding the impact of LiveOps events on LTV is difficult when only 3 or 4 LiveOps have been done. You can avoid having to wait by looking at peaks during the LiveOps event. "Slice" the LTV curve into smaller periods, define a validation cohort for each slice and calculate the impact on LTV over the period. Then normalize the impact and calculate the final improvement. Example chart here

01:12:21
💎 #
15

When evaluating the impact of LiveOps on LTV, do not forget to take into account the novelty effect: peaks tend to be higher during the first LiveOps events.

01:13:45
💎 #
16

For special offers LTV prediction models start with a rule-based system, then a probabilistic system and finally machine learning system. With the rule-based system it is less risky and more transparent which also helps you identify the impact of the changes you make.

01:21:55
The access to discussions on each resource is only available to premium members.

Growth Gems members discuss resources and their key insights (as well as other mobile growth topics) in the community. It's the perfect way to dig deeper, ask questions and get additional perspectives.
Upgrade to premium to:
  • Unlock access to key insights from over 150 mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the member-only community.
Upgrade Your Plan
The access to discussions on each resource is only available to premium members.

Growth Gems members discuss resources and their key insights (as well as other mobile growth topics) in the community. It's the perfect way to dig deeper, ask questions and get additional perspectives.
Become a member to:
  • Unlock access to gems from over 150 mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the member-only community.
Request Access

You need to be logged in the community to be able to see the discussion below.
You can also head over directly to this topic in the community

Notes for this resource are currently being transferred and will be available soon.

Session #1 - LTV predictions for Growth and UA activities


Intermediate and sophisticated approaches towards LTV measurement. Examples of custom and advanced solutions built to understand LTV of the users. What is easy, what is more challenging once you work on such solutions. How analytics and Growth teams can work together for final outcome. Presentation based on examples and case studies.


How to combat scaling issues


[💎 @10:15] Chart your creatives on a X axis and check your D3/D7/D28 ROAS to quickly spot outlier creatives (both good and bad) so you can act on that (by reallocating spend for example).

[💎 @11:29] A benchmark comparing ROAS (e.g. D7/D28) for each week (X axis) with success thresholds allows you to evaluate your UA strategies.


[💎 @13:54] Understanding the CPI to spend relationship is a key factor in understanding how you can scale your campaign.

[💎 @19:13] Questions you need to ask yourself to find the most profitable players (for LAL on Android for example - changes coming to iOS):

1. Is this group great at buying IAPS, do they do it frequently?

2. Is this group heavily engaged, does their engagement grow over time?

3. Do this group of players watch ads frequently?

Make sure you have a benchmark to compare these new LAL/audiences to.

[💎 @21:45] Do not create too many groups/segments of players. You need to make sure they are different so you can understand which group is better. It is not enough to segment based on how much they purchase, you need to use other attributes too.


[💎 @24:12] If your LTV curve looks like a step function with jumps, either your game is relying mainly on LiveOps offers (not ideal design) or the amount of payers/players is too low.

Improving IAP LTV




[💎 @30:50] Your special offers are great if you can increase revenue per user while minimizing the discount. The value is in the personalization and showing the right offer (with a relevant content and price) at the right time. Example of special offer delivery system.

Differences in LTV prediction calculations for hypercasual games vs. IAP based games?

Same model but you change the data. Leverage more ad attributes if you rely more on ads.


Session #2 - Developing an effective LTV model at the soft launch and keeping it valid further beyond.

We will follow the whole way of developing and maintaining an LTV model for our game starting from the very rough extrapolation models at the soft launch and step by step will reach accurate user-based Machine Learning models for mature products. Moreover, we will peek into the main obstacles on our way and how to overcome them.

Works with games with different monetization models

[💎 @42:58] Predicting LTV is different at different game stages: soft launch, some time after global launch or when the game is at maturity. See characteristics and suggested approach below.

  • Soft launch (uniform users because 1 or 2 geos)
  • Some time after global launch
  • Game maturity

LTV at soft launch


[💎 @52:00] In soft launch we do not have the whole LTV curve but we somehow need to calculate the lifetime length so product knowledge is crucial because you're extrapolating. You have to know:

  • Monetization limits (depth)
  • User behavior
  • Use different scenarios

Model validation

[💎 @54:39] The most important step in LTV model development is your LTV model and forecast validation. Always have a validation sample to test the model against, and it must be representative. Make sure you do not build the model to work especially well against your validation sample (i.e. "overfit").


LTV some time after global launch

[💎 @58:10] Some time after global launch you need different LTV models: country groups (Tier 1 vs. Tier 2 vs. Tier 3), acquisition sources and optimization types (Google Ads vs. video networks) and monetization types (in-apps vs. ad-based, or live ops vs. regular purchases).


[💎 @01:07:02] Always think about how the LTV model will be used. Example: LTV model for the UA team needs to be working with a small sample size so decisions can be made at the campaign level vs. LTV model for strategic decisions needs to be more accurate and can be more thorough.

Mature game

[💎 @01:10:18] If you are encountering issues when leveraging machine learning, build a quick model with rough "soft launch techniques" for quick validation. Have a few models using very limited amount of data so you can retrain the machine learning models as soon as possible.


[💎 @01:12:21] Understanding the LiveOps events impact on LTV is difficult when only 3 or 4 LiveOps have been done. You can avoid waiting by looking at peaks during the LiveOps event. "Slice" the LTV curve into smaller periods, define a validation cohort for each slice and calculate the impact on LTV over the period. Then normalize the impact and calculate the final improvement. Example chart below.

[💎 @01:13:45] When evaluating the impact of LiveOps on LTV, do not forget to take into account the novelty effect: peaks tend to be higher during the first LiveOps events.


Impact of iOS 14?

Ivan: 

  • Few of the important features to predict LTV won't be available anymore on iOS, like user origin (type of optimization). So they will rely on more robust but less accurate LTV models. The impact of iOS 14 is huge for optimization, not that big for LTV.

Robert

  • Still researching but there are still ways to do it. Android still possible at least. It's going to depend on the game.

In Google App Campaigns, is it possible to track D3/D7 ROAS at the creative level?

  • Robert: what he was showing was facebook, not sure if it's possible for Google App Campaigns.

Do you use reinforcement learning for offers?

  • Robert: for special ops it's important to bring as much uplift as possible. Built a few designs for it but didn't go through with it.

Optimizing LTV models conservatively vs. aggressively and taking on a bigger risk of error?

Robert

  • [💎 @01:21:55] For special offers LTV prediction models start with a rule-based system, then a probabilistic system and finally machine learning system. With the rule-based system it is less risky and more transparent which also helps you identify the impact of the changes you make.
  • Build models separately


Different models between markets/geos?

Ivan

  • Differences depend on the game and its maturity stage, especially if you do not have the same monetization model from one market/geo to another

LTV calculations for ad monetized games

Robert

  • Not different but you find the players that are the best at watching the ads
  • You can not be as confident with the prediction vs. IAP game because there are more things that you don't control (mediation, eCPMs etc. that are not reliable) and you're dealing with small monetization amounts
  • Need to monitor quickly and more frequently

Ivan

  • It's always hard regardless of the monetization model
  • With Ad monetized games you are working with more users typically vs. IAP users
  • You supplement the models with different features. For ad monetized games: engagement features

A/B testing for monetization

Robert

  • You only have a few months to change something typically so in order not to loose time they use a bayesian algorithm. 90% confidence but faster results.Choose your tests based on the expected impact on revenue.

Ivan

  • Usually you only have 2 months and you need to do your tests quickly and prioritize well. It often comes down to an expert's opinion.

Robert

  • Try to test the core of the hypothesis and test it in real time


The notes from this resource are only available to premium members.
↘ At this point, you know what to do ↙
Upgrade Your Plan

Session #1 - LTV predictions for Growth and UA activities


Intermediate and sophisticated approaches towards LTV measurement. Examples of custom and advanced solutions built to understand LTV of the users. What is easy, what is more challenging once you work on such solutions. How analytics and Growth teams can work together for final outcome. Presentation based on examples and case studies.


How to combat scaling issues


[💎 @10:15] Chart your creatives on a X axis and check your D3/D7/D28 ROAS to quickly spot outlier creatives (both good and bad) so you can act on that (by reallocating spend for example).

[💎 @11:29] A benchmark comparing ROAS (e.g. D7/D28) for each week (X axis) with success thresholds allows you to evaluate your UA strategies.


[💎 @13:54] Understanding the CPI to spend relationship is a key factor in understanding how you can scale your campaign.

[💎 @19:13] Questions you need to ask yourself to find the most profitable players (for LAL on Android for example - changes coming to iOS):

1. Is this group great at buying IAPS, do they do it frequently?

2. Is this group heavily engaged, does their engagement grow over time?

3. Do this group of players watch ads frequently?

Make sure you have a benchmark to compare these new LAL/audiences to.

[💎 @21:45] Do not create too many groups/segments of players. You need to make sure they are different so you can understand which group is better. It is not enough to segment based on how much they purchase, you need to use other attributes too.


[💎 @24:12] If your LTV curve looks like a step function with jumps, either your game is relying mainly on LiveOps offers (not ideal design) or the amount of payers/players is too low.

Improving IAP LTV




[💎 @30:50] Your special offers are great if you can increase revenue per user while minimizing the discount. The value is in the personalization and showing the right offer (with a relevant content and price) at the right time. Example of special offer delivery system.

Differences in LTV prediction calculations for hypercasual games vs. IAP based games?

Same model but you change the data. Leverage more ad attributes if you rely more on ads.


Session #2 - Developing an effective LTV model at the soft launch and keeping it valid further beyond.

We will follow the whole way of developing and maintaining an LTV model for our game starting from the very rough extrapolation models at the soft launch and step by step will reach accurate user-based Machine Learning models for mature products. Moreover, we will peek into the main obstacles on our way and how to overcome them.

Works with games with different monetization models

[💎 @42:58] Predicting LTV is different at different game stages: soft launch, some time after global launch or when the game is at maturity. See characteristics and suggested approach below.

  • Soft launch (uniform users because 1 or 2 geos)
  • Some time after global launch
  • Game maturity

LTV at soft launch


[💎 @52:00] In soft launch we do not have the whole LTV curve but we somehow need to calculate the lifetime length so product knowledge is crucial because you're extrapolating. You have to know:

  • Monetization limits (depth)
  • User behavior
  • Use different scenarios

Model validation

[💎 @54:39] The most important step in LTV model development is your LTV model and forecast validation. Always have a validation sample to test the model against, and it must be representative. Make sure you do not build the model to work especially well against your validation sample (i.e. "overfit").


LTV some time after global launch

[💎 @58:10] Some time after global launch you need different LTV models: country groups (Tier 1 vs. Tier 2 vs. Tier 3), acquisition sources and optimization types (Google Ads vs. video networks) and monetization types (in-apps vs. ad-based, or live ops vs. regular purchases).


[💎 @01:07:02] Always think about how the LTV model will be used. Example: LTV model for the UA team needs to be working with a small sample size so decisions can be made at the campaign level vs. LTV model for strategic decisions needs to be more accurate and can be more thorough.

Mature game

[💎 @01:10:18] If you are encountering issues when leveraging machine learning, build a quick model with rough "soft launch techniques" for quick validation. Have a few models using very limited amount of data so you can retrain the machine learning models as soon as possible.


[💎 @01:12:21] Understanding the LiveOps events impact on LTV is difficult when only 3 or 4 LiveOps have been done. You can avoid waiting by looking at peaks during the LiveOps event. "Slice" the LTV curve into smaller periods, define a validation cohort for each slice and calculate the impact on LTV over the period. Then normalize the impact and calculate the final improvement. Example chart below.

[💎 @01:13:45] When evaluating the impact of LiveOps on LTV, do not forget to take into account the novelty effect: peaks tend to be higher during the first LiveOps events.


Impact of iOS 14?

Ivan: 

  • Few of the important features to predict LTV won't be available anymore on iOS, like user origin (type of optimization). So they will rely on more robust but less accurate LTV models. The impact of iOS 14 is huge for optimization, not that big for LTV.

Robert

  • Still researching but there are still ways to do it. Android still possible at least. It's going to depend on the game.

In Google App Campaigns, is it possible to track D3/D7 ROAS at the creative level?

  • Robert: what he was showing was facebook, not sure if it's possible for Google App Campaigns.

Do you use reinforcement learning for offers?

  • Robert: for special ops it's important to bring as much uplift as possible. Built a few designs for it but didn't go through with it.

Optimizing LTV models conservatively vs. aggressively and taking on a bigger risk of error?

Robert

  • [💎 @01:21:55] For special offers LTV prediction models start with a rule-based system, then a probabilistic system and finally machine learning system. With the rule-based system it is less risky and more transparent which also helps you identify the impact of the changes you make.
  • Build models separately


Different models between markets/geos?

Ivan

  • Differences depend on the game and its maturity stage, especially if you do not have the same monetization model from one market/geo to another

LTV calculations for ad monetized games

Robert

  • Not different but you find the players that are the best at watching the ads
  • You can not be as confident with the prediction vs. IAP game because there are more things that you don't control (mediation, eCPMs etc. that are not reliable) and you're dealing with small monetization amounts
  • Need to monitor quickly and more frequently

Ivan

  • It's always hard regardless of the monetization model
  • With Ad monetized games you are working with more users typically vs. IAP users
  • You supplement the models with different features. For ad monetized games: engagement features

A/B testing for monetization

Robert

  • You only have a few months to change something typically so in order not to loose time they use a bayesian algorithm. 90% confidence but faster results.Choose your tests based on the expected impact on revenue.

Ivan

  • Usually you only have 2 months and you need to do your tests quickly and prioritize well. It often comes down to an expert's opinion.

Robert

  • Try to test the core of the hypothesis and test it in real time


The notes from this resource are only available to premium members.

The detailed notes taken for a resource are an easy way to see the gems in context to get a better understanding. They also include any relevant visuals from the source.
↘ At this point, you know what to do ↙
Request Access

Session #1 - LTV predictions for Growth and UA activities


Intermediate and sophisticated approaches towards LTV measurement. Examples of custom and advanced solutions built to understand LTV of the users. What is easy, what is more challenging once you work on such solutions. How analytics and Growth teams can work together for final outcome. Presentation based on examples and case studies.


How to combat scaling issues


[💎 @10:15] Chart your creatives on a X axis and check your D3/D7/D28 ROAS to quickly spot outlier creatives (both good and bad) so you can act on that (by reallocating spend for example).

[💎 @11:29] A benchmark comparing ROAS (e.g. D7/D28) for each week (X axis) with success thresholds allows you to evaluate your UA strategies.


[💎 @13:54] Understanding the CPI to spend relationship is a key factor in understanding how you can scale your campaign.

[💎 @19:13] Questions you need to ask yourself to find the most profitable players (for LAL on Android for example - changes coming to iOS):

1. Is this group great at buying IAPS, do they do it frequently?

2. Is this group heavily engaged, does their engagement grow over time?

3. Do this group of players watch ads frequently?

Make sure you have a benchmark to compare these new LAL/audiences to.

[💎 @21:45] Do not create too many groups/segments of players. You need to make sure they are different so you can understand which group is better. It is not enough to segment based on how much they purchase, you need to use other attributes too.


[💎 @24:12] If your LTV curve looks like a step function with jumps, either your game is relying mainly on LiveOps offers (not ideal design) or the amount of payers/players is too low.

Improving IAP LTV




[💎 @30:50] Your special offers are great if you can increase revenue per user while minimizing the discount. The value is in the personalization and showing the right offer (with a relevant content and price) at the right time. Example of special offer delivery system.

Differences in LTV prediction calculations for hypercasual games vs. IAP based games?

Same model but you change the data. Leverage more ad attributes if you rely more on ads.


Session #2 - Developing an effective LTV model at the soft launch and keeping it valid further beyond.

We will follow the whole way of developing and maintaining an LTV model for our game starting from the very rough extrapolation models at the soft launch and step by step will reach accurate user-based Machine Learning models for mature products. Moreover, we will peek into the main obstacles on our way and how to overcome them.

Works with games with different monetization models

[💎 @42:58] Predicting LTV is different at different game stages: soft launch, some time after global launch or when the game is at maturity. See characteristics and suggested approach below.

  • Soft launch (uniform users because 1 or 2 geos)
  • Some time after global launch
  • Game maturity

LTV at soft launch


[💎 @52:00] In soft launch we do not have the whole LTV curve but we somehow need to calculate the lifetime length so product knowledge is crucial because you're extrapolating. You have to know:

  • Monetization limits (depth)
  • User behavior
  • Use different scenarios

Model validation

[💎 @54:39] The most important step in LTV model development is your LTV model and forecast validation. Always have a validation sample to test the model against, and it must be representative. Make sure you do not build the model to work especially well against your validation sample (i.e. "overfit").


LTV some time after global launch

[💎 @58:10] Some time after global launch you need different LTV models: country groups (Tier 1 vs. Tier 2 vs. Tier 3), acquisition sources and optimization types (Google Ads vs. video networks) and monetization types (in-apps vs. ad-based, or live ops vs. regular purchases).


[💎 @01:07:02] Always think about how the LTV model will be used. Example: LTV model for the UA team needs to be working with a small sample size so decisions can be made at the campaign level vs. LTV model for strategic decisions needs to be more accurate and can be more thorough.

Mature game

[💎 @01:10:18] If you are encountering issues when leveraging machine learning, build a quick model with rough "soft launch techniques" for quick validation. Have a few models using very limited amount of data so you can retrain the machine learning models as soon as possible.


[💎 @01:12:21] Understanding the LiveOps events impact on LTV is difficult when only 3 or 4 LiveOps have been done. You can avoid waiting by looking at peaks during the LiveOps event. "Slice" the LTV curve into smaller periods, define a validation cohort for each slice and calculate the impact on LTV over the period. Then normalize the impact and calculate the final improvement. Example chart below.

[💎 @01:13:45] When evaluating the impact of LiveOps on LTV, do not forget to take into account the novelty effect: peaks tend to be higher during the first LiveOps events.


Impact of iOS 14?

Ivan: 

  • Few of the important features to predict LTV won't be available anymore on iOS, like user origin (type of optimization). So they will rely on more robust but less accurate LTV models. The impact of iOS 14 is huge for optimization, not that big for LTV.

Robert

  • Still researching but there are still ways to do it. Android still possible at least. It's going to depend on the game.

In Google App Campaigns, is it possible to track D3/D7 ROAS at the creative level?

  • Robert: what he was showing was facebook, not sure if it's possible for Google App Campaigns.

Do you use reinforcement learning for offers?

  • Robert: for special ops it's important to bring as much uplift as possible. Built a few designs for it but didn't go through with it.

Optimizing LTV models conservatively vs. aggressively and taking on a bigger risk of error?

Robert

  • [💎 @01:21:55] For special offers LTV prediction models start with a rule-based system, then a probabilistic system and finally machine learning system. With the rule-based system it is less risky and more transparent which also helps you identify the impact of the changes you make.
  • Build models separately


Different models between markets/geos?

Ivan

  • Differences depend on the game and its maturity stage, especially if you do not have the same monetization model from one market/geo to another

LTV calculations for ad monetized games

Robert

  • Not different but you find the players that are the best at watching the ads
  • You can not be as confident with the prediction vs. IAP game because there are more things that you don't control (mediation, eCPMs etc. that are not reliable) and you're dealing with small monetization amounts
  • Need to monitor quickly and more frequently

Ivan

  • It's always hard regardless of the monetization model
  • With Ad monetized games you are working with more users typically vs. IAP users
  • You supplement the models with different features. For ad monetized games: engagement features

A/B testing for monetization

Robert

  • You only have a few months to change something typically so in order not to loose time they use a bayesian algorithm. 90% confidence but faster results.Choose your tests based on the expected impact on revenue.

Ivan

  • Usually you only have 2 months and you need to do your tests quickly and prioritize well. It often comes down to an expert's opinion.

Robert

  • Try to test the core of the hypothesis and test it in real time