The ABCs of Mobile A/B Testing

Item not bookmarked
Resource bookmarked
Bookmarking...
Review star
Review star
Review star
Review star
Review star
💎  x
3

Lina Danilchik (Marketing Manager at Splitmetrics - A/B testing tool) and Maggie Ngai (Senior ASO Consultant - Phiture) talk about Mobile A/B testing best practices, pretotyping (and why it's a good alternative to prototyping).

Source:
The ABCs of Mobile A/B Testing
(no direct link to watch/listen)
(direct link to watch/listen)
Type:
Webinar
Publication date:
March 25, 2020
Added to the Vault on:
March 27, 2020
These insights were shared through the free Growth Gems newsletter.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
💎 #
1

7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app). 

42:00
💎 #
2

If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas). 

42:50
💎 #
3

Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc. 

44:10
The "gems" from this resource are only available to premium members.
  • Unlock access to gems from over 130 mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the members-only community.
Upgrade Your Plan
💎 #
1

7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app). 

42:00
💎 #
2

If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas). 

42:50
💎 #
3

Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc. 

44:10
The "gems" from this resource are only available to premium members.

Gems are the key bite-size insights "mined" from a specific mobile marketing resource, like a webinar, a panel or a podcast.
They allow you to save time by grasping the most important information in a couple of minutes, and also each include the timestamp from the source.

Become a member to:
  • Unlock access to gems from over 130 mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the members-only community.
Request Access
💎 #
1

7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app). 

42:00
💎 #
2

If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas). 

42:50
💎 #
3

Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc. 

44:10
The access to discussions on each resource is only available to premium members.

Growth Gems members discuss resources and their key insights (as well as other mobile growth topics) in the community. It's the perfect way to dig deeper, ask questions and get additional perspectives.
Upgrade to premium to:
  • Unlock access to key insights from over 130 mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the members-only community.
Upgrade Your Plan
The access to discussions on each resource is only available to premium members.

Growth Gems members discuss resources and their key insights (as well as other mobile growth topics) in the community. It's the perfect way to dig deeper, ask questions and get additional perspectives.
Become a member to:
  • Unlock access to gems from over 130 mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the members-only community.
Request Access

You need to be logged in the community to be able to see the discussion below.
You can also head over directly to this topic in the community

Notes for this resource are currently being transferred and will be available soon.

What is pretotyping?

A pretotype is an image that conveys the idea of a product and takes only a few hours.


What can serve as pretotypes: app title, icon, description and screenshots.

You can test your app ideas via a pretotype with an A/B testing platform like Splitmetrics. You can then decide if your should build it but also compare different app concepts or features and select what converts best.


Mobile A/B testing can also help you decide on localization assets.


Main metrics to rely on when running tests on SplitMetrics

  • Average time on page
  • Click to Install rate (visitors tapping install button)
  • Engagement rate (visitors who interact with the page)


General approach to A/B testing

Regardless of the tool used (Splitmetrics, Google Experiments, ASA Creative Sets) Phiture never leaves the CRO framework.


  1. Research on what you should/could test - user research, agency, internet, etc.
  2. Draw hypothesis - so you can understand why certain things happen
  3. Create assets
  4. Run test
  5. Look at the conversion rate and whether people download the app more or not. Can also look at down the funnel events but #1 is looking at downloads
  6. Go back to step #1


Things to test


  • Icon (for games, very important to feature the right game characters)
  • Screenshots - messaging tests. look and feel, etc.
  • Short description on Google Play
  • Etc.


Be careful with "risky" elements (e.g. feature that does not exist yet). The short description might be a better place to test this.

However you don't want to deceive users and create mistrust. You need to have consistency between listing and the actual app. So keep volume just enough to get statistical significance: the goal is not to get downloads.


Biggest mistakes in Mobile A/B Testing

  1. Finishing an experiment before you get the right amount of traffic
  2. Finishing an experiment before 7 days have passed (results can depend on day - e.g. games on weekend, financial apps on Monday, etc.)
  3. Testing too small changes in design - test bigger things!
  4. Using a banner ad that has the same design as one of the App Store visual assets (people might instinctively tap on the visual that looks the same once they get to the testing page)
  5. Testing several hypotheses at once - makes it too hard to analyze and understand what affects the results, especially if there are small changes
  6. Misinterpreting the situation when running an A/A test but you get a winner
  7. Getting upset when a new variation loses to the current one


Q&A

How to judge the results?

Before getting started, calculate the sample size that you need to make a test so you have enough traffic. When looking at the results you also need to have a significant difference in the conversion rate.


When to test?

[💎@42:00] 7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).

[💎@42:50] If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).


[💎@44:10] Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc.


What is a good conversion rate?

Check benchmarks online but in general a 30% conversion rate is considered good.

Some app publishers manage to reach +60% conversion uplift.


Using ASA Creative Sets

You can test 2 sets of 3 screenshots and look at TTR, CVR. It is not a fair A/B testing. You can test with your brand keywords or your generic keywords. Check out this article.



The notes from this resource are only available to premium members.
↘ At this point, you know what to do ↙
Upgrade Your Plan

What is pretotyping?

A pretotype is an image that conveys the idea of a product and takes only a few hours.


What can serve as pretotypes: app title, icon, description and screenshots.

You can test your app ideas via a pretotype with an A/B testing platform like Splitmetrics. You can then decide if your should build it but also compare different app concepts or features and select what converts best.


Mobile A/B testing can also help you decide on localization assets.


Main metrics to rely on when running tests on SplitMetrics

  • Average time on page
  • Click to Install rate (visitors tapping install button)
  • Engagement rate (visitors who interact with the page)


General approach to A/B testing

Regardless of the tool used (Splitmetrics, Google Experiments, ASA Creative Sets) Phiture never leaves the CRO framework.


  1. Research on what you should/could test - user research, agency, internet, etc.
  2. Draw hypothesis - so you can understand why certain things happen
  3. Create assets
  4. Run test
  5. Look at the conversion rate and whether people download the app more or not. Can also look at down the funnel events but #1 is looking at downloads
  6. Go back to step #1


Things to test


  • Icon (for games, very important to feature the right game characters)
  • Screenshots - messaging tests. look and feel, etc.
  • Short description on Google Play
  • Etc.


Be careful with "risky" elements (e.g. feature that does not exist yet). The short description might be a better place to test this.

However you don't want to deceive users and create mistrust. You need to have consistency between listing and the actual app. So keep volume just enough to get statistical significance: the goal is not to get downloads.


Biggest mistakes in Mobile A/B Testing

  1. Finishing an experiment before you get the right amount of traffic
  2. Finishing an experiment before 7 days have passed (results can depend on day - e.g. games on weekend, financial apps on Monday, etc.)
  3. Testing too small changes in design - test bigger things!
  4. Using a banner ad that has the same design as one of the App Store visual assets (people might instinctively tap on the visual that looks the same once they get to the testing page)
  5. Testing several hypotheses at once - makes it too hard to analyze and understand what affects the results, especially if there are small changes
  6. Misinterpreting the situation when running an A/A test but you get a winner
  7. Getting upset when a new variation loses to the current one


Q&A

How to judge the results?

Before getting started, calculate the sample size that you need to make a test so you have enough traffic. When looking at the results you also need to have a significant difference in the conversion rate.


When to test?

[💎@42:00] 7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).

[💎@42:50] If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).


[💎@44:10] Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc.


What is a good conversion rate?

Check benchmarks online but in general a 30% conversion rate is considered good.

Some app publishers manage to reach +60% conversion uplift.


Using ASA Creative Sets

You can test 2 sets of 3 screenshots and look at TTR, CVR. It is not a fair A/B testing. You can test with your brand keywords or your generic keywords. Check out this article.



The notes from this resource are only available to premium members.

The detailed notes taken for a resource are an easy way to see the gems in context to get a better understanding. They also include any relevant visuals from the source.
↘ At this point, you know what to do ↙
Request Access

What is pretotyping?

A pretotype is an image that conveys the idea of a product and takes only a few hours.


What can serve as pretotypes: app title, icon, description and screenshots.

You can test your app ideas via a pretotype with an A/B testing platform like Splitmetrics. You can then decide if your should build it but also compare different app concepts or features and select what converts best.


Mobile A/B testing can also help you decide on localization assets.


Main metrics to rely on when running tests on SplitMetrics

  • Average time on page
  • Click to Install rate (visitors tapping install button)
  • Engagement rate (visitors who interact with the page)


General approach to A/B testing

Regardless of the tool used (Splitmetrics, Google Experiments, ASA Creative Sets) Phiture never leaves the CRO framework.


  1. Research on what you should/could test - user research, agency, internet, etc.
  2. Draw hypothesis - so you can understand why certain things happen
  3. Create assets
  4. Run test
  5. Look at the conversion rate and whether people download the app more or not. Can also look at down the funnel events but #1 is looking at downloads
  6. Go back to step #1


Things to test


  • Icon (for games, very important to feature the right game characters)
  • Screenshots - messaging tests. look and feel, etc.
  • Short description on Google Play
  • Etc.


Be careful with "risky" elements (e.g. feature that does not exist yet). The short description might be a better place to test this.

However you don't want to deceive users and create mistrust. You need to have consistency between listing and the actual app. So keep volume just enough to get statistical significance: the goal is not to get downloads.


Biggest mistakes in Mobile A/B Testing

  1. Finishing an experiment before you get the right amount of traffic
  2. Finishing an experiment before 7 days have passed (results can depend on day - e.g. games on weekend, financial apps on Monday, etc.)
  3. Testing too small changes in design - test bigger things!
  4. Using a banner ad that has the same design as one of the App Store visual assets (people might instinctively tap on the visual that looks the same once they get to the testing page)
  5. Testing several hypotheses at once - makes it too hard to analyze and understand what affects the results, especially if there are small changes
  6. Misinterpreting the situation when running an A/A test but you get a winner
  7. Getting upset when a new variation loses to the current one


Q&A

How to judge the results?

Before getting started, calculate the sample size that you need to make a test so you have enough traffic. When looking at the results you also need to have a significant difference in the conversion rate.


When to test?

[💎@42:00] 7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).

[💎@42:50] If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).


[💎@44:10] Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc.


What is a good conversion rate?

Check benchmarks online but in general a 30% conversion rate is considered good.

Some app publishers manage to reach +60% conversion uplift.


Using ASA Creative Sets

You can test 2 sets of 3 screenshots and look at TTR, CVR. It is not a fair A/B testing. You can test with your brand keywords or your generic keywords. Check out this article.