Karan Tibdewal (Growth Consultant at Phiture - mobile growth consultancy) talks about how you can level up your A/B testing game to maximise impact and learnings
To get to the point where more testing = more growth being true you need to have a thorough process in place.
3 key goals of the testing framework are:
1. Communicate ideas from all teams and prioritize based on company objectives 2. Understand and define leading and lagging metrics
3. Centralize test results and communicate next steps
Use slides for your experiments: excel files are good for internal use but slides are better because they help get the buy-in from different teams and get them involved.
When it comes to goal metrics, you do not want to only optimize for revenue: you want "leading metrics" that feed in to revenue because there are a lot of variables that go into optimizing for revenue.
To know which ideas to prioritize, measure the Impact score with Impact = Reach x Relevance x Frequency.
Set up experiments for success by planning ahead of time and using a sample size calculator tool like the one from Optimizely (link here).
You should not have a "win or lose" mentality. Once you have a significant positive impact you want to scale up the test (apply to 100%) then you want to start iterating on it. Double down on high impact tests and keep iterating.
After 2 or 3 variations of a test idea that does not prove a significant positive impact, do not get too attached to your idea and stop iterating: it is not meant to be.
Do a reach audit where you assess how many tests you could run with your user base with your current engagement rate. Example: don't just look at how many people will receive that email and assume a 10% conversion rate. Instead, look at your past performance, look at how many users you can reach and use calculator to figure out time to reach statistical significance.
Onboarding and activation funnels are the main areas of improvement for almost all of the apps (even 5+ years apps!).
How long to run tests for? A week is often enough but it of course depends on volume. Usually the absolute difference between variants should be more than 200-300 to get proper test results (if you are very small). Do not go for tests that are more than 1-1.5 months.
How often do you revisit experiments? Once you have an experiment in place and you have maximized/iterated on it, it is good to have quarterly reviews.
To get to the point where more testing = more growth being true you need to have a thorough process in place.
3 key goals of the testing framework are:
1. Communicate ideas from all teams and prioritize based on company objectives 2. Understand and define leading and lagging metrics
3. Centralize test results and communicate next steps
Use slides for your experiments: excel files are good for internal use but slides are better because they help get the buy-in from different teams and get them involved.
When it comes to goal metrics, you do not want to only optimize for revenue: you want "leading metrics" that feed in to revenue because there are a lot of variables that go into optimizing for revenue.
To know which ideas to prioritize, measure the Impact score with Impact = Reach x Relevance x Frequency.
Set up experiments for success by planning ahead of time and using a sample size calculator tool like the one from Optimizely (link here).
You should not have a "win or lose" mentality. Once you have a significant positive impact you want to scale up the test (apply to 100%) then you want to start iterating on it. Double down on high impact tests and keep iterating.
After 2 or 3 variations of a test idea that does not prove a significant positive impact, do not get too attached to your idea and stop iterating: it is not meant to be.
Do a reach audit where you assess how many tests you could run with your user base with your current engagement rate. Example: don't just look at how many people will receive that email and assume a 10% conversion rate. Instead, look at your past performance, look at how many users you can reach and use calculator to figure out time to reach statistical significance.
Onboarding and activation funnels are the main areas of improvement for almost all of the apps (even 5+ years apps!).
How long to run tests for? A week is often enough but it of course depends on volume. Usually the absolute difference between variants should be more than 200-300 to get proper test results (if you are very small). Do not go for tests that are more than 1-1.5 months.
How often do you revisit experiments? Once you have an experiment in place and you have maximized/iterated on it, it is good to have quarterly reviews.
To get to the point where more testing = more growth being true you need to have a thorough process in place.
3 key goals of the testing framework are:
1. Communicate ideas from all teams and prioritize based on company objectives 2. Understand and define leading and lagging metrics
3. Centralize test results and communicate next steps
Use slides for your experiments: excel files are good for internal use but slides are better because they help get the buy-in from different teams and get them involved.
When it comes to goal metrics, you do not want to only optimize for revenue: you want "leading metrics" that feed in to revenue because there are a lot of variables that go into optimizing for revenue.
To know which ideas to prioritize, measure the Impact score with Impact = Reach x Relevance x Frequency.
Set up experiments for success by planning ahead of time and using a sample size calculator tool like the one from Optimizely (link here).
You should not have a "win or lose" mentality. Once you have a significant positive impact you want to scale up the test (apply to 100%) then you want to start iterating on it. Double down on high impact tests and keep iterating.
After 2 or 3 variations of a test idea that does not prove a significant positive impact, do not get too attached to your idea and stop iterating: it is not meant to be.
Do a reach audit where you assess how many tests you could run with your user base with your current engagement rate. Example: don't just look at how many people will receive that email and assume a 10% conversion rate. Instead, look at your past performance, look at how many users you can reach and use calculator to figure out time to reach statistical significance.
Onboarding and activation funnels are the main areas of improvement for almost all of the apps (even 5+ years apps!).
How long to run tests for? A week is often enough but it of course depends on volume. Usually the absolute difference between variants should be more than 200-300 to get proper test results (if you are very small). Do not go for tests that are more than 1-1.5 months.
How often do you revisit experiments? Once you have an experiment in place and you have maximized/iterated on it, it is good to have quarterly reviews.
Notes for this resource are currently being transferred and will be available soon.
Different levels depending on company:
This talk focuses on Level 2
[💎@14:12] 3 key goals of the testing framework are :
Different organizations set up their team structures in different ways: CRM doesn't know what Product is doing vs. Metric-driven organization.
Framework allows for iterative learnings, even if you do not do it step by step every time.
While going through the framework, there are 3 key output documents:
Main ways to come up with hypothesis
Initial ideation session to create a backlog (SoundCloud illustrative example)
The make or break of the testing is how you specify and prioritize ideas. You need to bring ideas down to reality in the Campaign Backlog Document.
[💎@22:48] When it comes to goal metrics, you do not want to only optimize for revenue: you want "leading metrics" that feed in to revenue because there are a lot of variables that go into optimizing for revenue.
Dependencies is a crucial part because you might have good ideas but team might not have time or resources to work on these ideas.
[💎@24:15] To know which ideas to prioritize, measure the Impact score with Impact = Reach x Relevance x Frequency.
Scale of 1 to 5
Resources required to push a test live is a critical element.
Measurement - Campaign and Global Controls
How are your campaigns set up, do you have the control groups in place. You do not want to just test variants, you also need a control group.
Besides the control group you want to start with a 50/50 split test because it gives you more direction.
The master control group size depends on your company size (the bigger the user base the smaller % you can use) and can allow you to prove the worth of a team.
Running valid experiments
Checklist to keep in mind
[💎@28:27] Set up experiments for success by planning ahead of time and using a sample size calculator tool like the one from Optimizely (link here).
You might want to give up a bit on significance to get more speed. In this case you have less precise test but can still get direction from results.
Your "leading indicator metrics" (e.g. subscription purchase, app open, outbound call) feed into your core metrics (W1/M1 retention, increase of power users).
[💎@31:20] You should not have a "win or lose" mentality. Once you have a significant positive impact you want to scale it (apply to 100%) then you want to start iterating on it. Double down on high impact tests and keep iterating.
[💎@32:05] After 2 or 3 variations of an idea that does not prove a significant positive impact, do not get too attached to your idea and stop iterating: it is not meant to be.
This is the most manual but the most vital when it comes to showing what you have worked on and getting buy-in from teams.
It helps when it is more visual
Different levels depending on company:
This talk focuses on Level 2
[💎@14:12] 3 key goals of the testing framework are :
Different organizations set up their team structures in different ways: CRM doesn't know what Product is doing vs. Metric-driven organization.
Framework allows for iterative learnings, even if you do not do it step by step every time.
While going through the framework, there are 3 key output documents:
Main ways to come up with hypothesis
Initial ideation session to create a backlog (SoundCloud illustrative example)
The make or break of the testing is how you specify and prioritize ideas. You need to bring ideas down to reality in the Campaign Backlog Document.
[💎@22:48] When it comes to goal metrics, you do not want to only optimize for revenue: you want "leading metrics" that feed in to revenue because there are a lot of variables that go into optimizing for revenue.
Dependencies is a crucial part because you might have good ideas but team might not have time or resources to work on these ideas.
[💎@24:15] To know which ideas to prioritize, measure the Impact score with Impact = Reach x Relevance x Frequency.
Scale of 1 to 5
Resources required to push a test live is a critical element.
Measurement - Campaign and Global Controls
How are your campaigns set up, do you have the control groups in place. You do not want to just test variants, you also need a control group.
Besides the control group you want to start with a 50/50 split test because it gives you more direction.
The master control group size depends on your company size (the bigger the user base the smaller % you can use) and can allow you to prove the worth of a team.
Running valid experiments
Checklist to keep in mind
[💎@28:27] Set up experiments for success by planning ahead of time and using a sample size calculator tool like the one from Optimizely (link here).
You might want to give up a bit on significance to get more speed. In this case you have less precise test but can still get direction from results.
Your "leading indicator metrics" (e.g. subscription purchase, app open, outbound call) feed into your core metrics (W1/M1 retention, increase of power users).
[💎@31:20] You should not have a "win or lose" mentality. Once you have a significant positive impact you want to scale it (apply to 100%) then you want to start iterating on it. Double down on high impact tests and keep iterating.
[💎@32:05] After 2 or 3 variations of an idea that does not prove a significant positive impact, do not get too attached to your idea and stop iterating: it is not meant to be.
This is the most manual but the most vital when it comes to showing what you have worked on and getting buy-in from teams.
It helps when it is more visual
Different levels depending on company:
This talk focuses on Level 2
[💎@14:12] 3 key goals of the testing framework are :
Different organizations set up their team structures in different ways: CRM doesn't know what Product is doing vs. Metric-driven organization.
Framework allows for iterative learnings, even if you do not do it step by step every time.
While going through the framework, there are 3 key output documents:
Main ways to come up with hypothesis
Initial ideation session to create a backlog (SoundCloud illustrative example)
The make or break of the testing is how you specify and prioritize ideas. You need to bring ideas down to reality in the Campaign Backlog Document.
[💎@22:48] When it comes to goal metrics, you do not want to only optimize for revenue: you want "leading metrics" that feed in to revenue because there are a lot of variables that go into optimizing for revenue.
Dependencies is a crucial part because you might have good ideas but team might not have time or resources to work on these ideas.
[💎@24:15] To know which ideas to prioritize, measure the Impact score with Impact = Reach x Relevance x Frequency.
Scale of 1 to 5
Resources required to push a test live is a critical element.
Measurement - Campaign and Global Controls
How are your campaigns set up, do you have the control groups in place. You do not want to just test variants, you also need a control group.
Besides the control group you want to start with a 50/50 split test because it gives you more direction.
The master control group size depends on your company size (the bigger the user base the smaller % you can use) and can allow you to prove the worth of a team.
Running valid experiments
Checklist to keep in mind
[💎@28:27] Set up experiments for success by planning ahead of time and using a sample size calculator tool like the one from Optimizely (link here).
You might want to give up a bit on significance to get more speed. In this case you have less precise test but can still get direction from results.
Your "leading indicator metrics" (e.g. subscription purchase, app open, outbound call) feed into your core metrics (W1/M1 retention, increase of power users).
[💎@31:20] You should not have a "win or lose" mentality. Once you have a significant positive impact you want to scale it (apply to 100%) then you want to start iterating on it. Double down on high impact tests and keep iterating.
[💎@32:05] After 2 or 3 variations of an idea that does not prove a significant positive impact, do not get too attached to your idea and stop iterating: it is not meant to be.
This is the most manual but the most vital when it comes to showing what you have worked on and getting buy-in from teams.
It helps when it is more visual
