Why the Control Always Wins and Facebook Creative Testing

💎  x
12

Brian Bowman (CEO of Consumer Acquisition) share the company's realization around why it is so difficult to beat the control video as well as the new creative testing process that resulted from that.

Type:
Webinar
Publication date:
August 13, 2020
Added to Growth Gems on:
August 23, 2020
💎 #
1

Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it

02:40
💎 #
2

It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.

04:12
💎 #
3

Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.

04:48
💎 #
4

CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)

05:41
💎 #
5

IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).

06;40
💎 #
6

Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

08:08
💎 #
7

For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.

09:20
💎 #
8

If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.

10:20
💎 #
9

Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.

12:06
💎 #
10

Testing creatives for ROAS in South Africa is a good proxy for some titles.

13:05
💎 #
11

 Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase

13:45
💎 #
12

ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.

14:50
This is premium content. Upgrade your plan in your profile to get access.
💎 #
1

Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it

02:40
💎 #
2

It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.

04:12
💎 #
3

Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.

04:48
💎 #
4

CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)

05:41
💎 #
5

IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).

06;40
💎 #
6

Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

08:08
💎 #
7

For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.

09:20
💎 #
8

If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.

10:20
💎 #
9

Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.

12:06
💎 #
10

Testing creatives for ROAS in South Africa is a good proxy for some titles.

13:05
💎 #
11

 Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase

13:45
💎 #
12

ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.

14:50
This content is available to Growth Gems premium members only.
Apply Now
💎 #
1

Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it

02:40
💎 #
2

It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.

04:12
💎 #
3

Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.

04:48
💎 #
4

CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)

05:41
💎 #
5

IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).

06;40
💎 #
6

Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

08:08
💎 #
7

For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.

09:20
💎 #
8

If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.

10:20
💎 #
9

Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.

12:06
💎 #
10

Testing creatives for ROAS in South Africa is a good proxy for some titles.

13:05
💎 #
11

 Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase

13:45
💎 #
12

ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.

14:50
This is premium content. Upgrade your plan in your profile to get access.
This content is available to Growth Gems premium members only.
Apply Now
You need to be logged in the community to see the discussion below. You can also head over to
this topic in the community
This is premium content. Upgrade your plan in your profile to get access.
This content is available to Growth Gems premium members only.
Apply Now