Lina Danilchik (Marketing Manager at Splitmetrics - A/B testing tool) and Maggie Ngai (Senior ASO Consultant - Phiture) talk about Mobile A/B testing best practices, pretotyping (and why it's a good alternative to prototyping).
7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).
If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).
7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).
If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).
7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).
If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).
Notes for this resource are currently being transferred and will be available soon.
A pretotype is an image that conveys the idea of a product and takes only a few hours.
What can serve as pretotypes: app title, icon, description and screenshots.
You can test your app ideas via a pretotype with an A/B testing platform like Splitmetrics. You can then decide if your should build it but also compare different app concepts or features and select what converts best.
Mobile A/B testing can also help you decide on localization assets.
Regardless of the tool used (Splitmetrics, Google Experiments, ASA Creative Sets) Phiture never leaves the CRO framework.
Be careful with "risky" elements (e.g. feature that does not exist yet). The short description might be a better place to test this.
However you don't want to deceive users and create mistrust. You need to have consistency between listing and the actual app. So keep volume just enough to get statistical significance: the goal is not to get downloads.
How to judge the results?
Before getting started, calculate the sample size that you need to make a test so you have enough traffic. When looking at the results you also need to have a significant difference in the conversion rate.
When to test?
[💎@42:00] 7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).
[💎@42:50] If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).
[💎@44:10] Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc.
What is a good conversion rate?
Check benchmarks online but in general a 30% conversion rate is considered good.
Some app publishers manage to reach +60% conversion uplift.
Using ASA Creative Sets
You can test 2 sets of 3 screenshots and look at TTR, CVR. It is not a fair A/B testing. You can test with your brand keywords or your generic keywords. Check out this article.
A pretotype is an image that conveys the idea of a product and takes only a few hours.
What can serve as pretotypes: app title, icon, description and screenshots.
You can test your app ideas via a pretotype with an A/B testing platform like Splitmetrics. You can then decide if your should build it but also compare different app concepts or features and select what converts best.
Mobile A/B testing can also help you decide on localization assets.
Regardless of the tool used (Splitmetrics, Google Experiments, ASA Creative Sets) Phiture never leaves the CRO framework.
Be careful with "risky" elements (e.g. feature that does not exist yet). The short description might be a better place to test this.
However you don't want to deceive users and create mistrust. You need to have consistency between listing and the actual app. So keep volume just enough to get statistical significance: the goal is not to get downloads.
How to judge the results?
Before getting started, calculate the sample size that you need to make a test so you have enough traffic. When looking at the results you also need to have a significant difference in the conversion rate.
When to test?
[💎@42:00] 7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).
[💎@42:50] If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).
[💎@44:10] Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc.
What is a good conversion rate?
Check benchmarks online but in general a 30% conversion rate is considered good.
Some app publishers manage to reach +60% conversion uplift.
Using ASA Creative Sets
You can test 2 sets of 3 screenshots and look at TTR, CVR. It is not a fair A/B testing. You can test with your brand keywords or your generic keywords. Check out this article.
A pretotype is an image that conveys the idea of a product and takes only a few hours.
What can serve as pretotypes: app title, icon, description and screenshots.
You can test your app ideas via a pretotype with an A/B testing platform like Splitmetrics. You can then decide if your should build it but also compare different app concepts or features and select what converts best.
Mobile A/B testing can also help you decide on localization assets.
Regardless of the tool used (Splitmetrics, Google Experiments, ASA Creative Sets) Phiture never leaves the CRO framework.
Be careful with "risky" elements (e.g. feature that does not exist yet). The short description might be a better place to test this.
However you don't want to deceive users and create mistrust. You need to have consistency between listing and the actual app. So keep volume just enough to get statistical significance: the goal is not to get downloads.
How to judge the results?
Before getting started, calculate the sample size that you need to make a test so you have enough traffic. When looking at the results you also need to have a significant difference in the conversion rate.
When to test?
[💎@42:00] 7 days can be a good cycle for tests. However at Phiture they usually consider the "business cycle" which will depend on the app (e.g. a month for a diet planning app).
[💎@42:50] If you are running messaging tests, do not run them on days or times where there is seasonality (e.g. music app at Christmas).
[💎@44:10] Phiture has an experiment roadmap for at least 6 months: hypothesis, where to test, etc.
What is a good conversion rate?
Check benchmarks online but in general a 30% conversion rate is considered good.
Some app publishers manage to reach +60% conversion uplift.
Using ASA Creative Sets
You can test 2 sets of 3 screenshots and look at TTR, CVR. It is not a fair A/B testing. You can test with your brand keywords or your generic keywords. Check out this article.
