Brian Bowman (CEO of Consumer Acquisition) share the company's realization around why it is so difficult to beat the control video as well as the new creative testing process that resulted from that.
Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it
It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.
Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.
CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)
IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).
Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.
For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.
If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.
Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.
Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase
ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.
Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it
It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.
Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.
CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)
IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).
Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.
For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.
If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.
Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.
Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase
ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.
Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it
It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.
Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.
CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)
IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).
Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.
For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.
If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.
Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.
Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase
ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.
Notes for this resource are currently being transferred and will be available soon.
Consumer Acquisition tested 60 different ads for a client and were expecting at least 2/3 winners, but none were able to beat the control.
This got them to duplicate the control several times and test all of the clones. One of the clones actually got an increased CPM.
The dog is replacing the client's ad for confidentiality reasons
[💎 @02:40] Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it.
[💎 @04:12] It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.
[💎 @04:48] Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.
Based on these learnings, Consumer Acquisition changed their creative testing process.
[💎 @05:41] CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)
[💎 @06:40] IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).
[💎 @08:08] Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.
[💎 @09:20] For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.
[💎 @10:20] If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.
[💎 @12:06] Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.
[💎 @13:05] Testing creatives for ROAS in South Africa is a good proxy for some titles.
[💎 @13:45] Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
[💎 @14:50] ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.
White papers from Consumer Acquisition: https://www.consumeracquisition.com/whitepapers/
Consumer Acquisition tested 60 different ads for a client and were expecting at least 2/3 winners, but none were able to beat the control.
This got them to duplicate the control several times and test all of the clones. One of the clones actually got an increased CPM.
The dog is replacing the client's ad for confidentiality reasons
[💎 @02:40] Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it.
[💎 @04:12] It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.
[💎 @04:48] Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.
Based on these learnings, Consumer Acquisition changed their creative testing process.
[💎 @05:41] CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)
[💎 @06:40] IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).
[💎 @08:08] Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.
[💎 @09:20] For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.
[💎 @10:20] If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.
[💎 @12:06] Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.
[💎 @13:05] Testing creatives for ROAS in South Africa is a good proxy for some titles.
[💎 @13:45] Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
[💎 @14:50] ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.
White papers from Consumer Acquisition: https://www.consumeracquisition.com/whitepapers/
Consumer Acquisition tested 60 different ads for a client and were expecting at least 2/3 winners, but none were able to beat the control.
This got them to duplicate the control several times and test all of the clones. One of the clones actually got an increased CPM.
The dog is replacing the client's ad for confidentiality reasons
[💎 @02:40] Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it.
[💎 @04:12] It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.
[💎 @04:48] Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.
Based on these learnings, Consumer Acquisition changed their creative testing process.
[💎 @05:41] CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)
[💎 @06:40] IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).
[💎 @08:08] Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.
[💎 @09:20] For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.
[💎 @10:20] If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.
[💎 @12:06] Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.
[💎 @13:05] Testing creatives for ROAS in South Africa is a good proxy for some titles.
[💎 @13:45] Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
[💎 @14:50] ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.
White papers from Consumer Acquisition: https://www.consumeracquisition.com/whitepapers/
