Why the Control Always Wins and Facebook Creative Testing

Item not bookmarked
Resource bookmarked
Bookmarking...
⛏️
Guest Miner:
Sylvain Gauchet
Review star
Review star
Review star
Review star
Review star
💎  x
12

Brian Bowman (CEO of Consumer Acquisition) share the company's realization around why it is so difficult to beat the control video as well as the new creative testing process that resulted from that.

Source:
Why the Control Always Wins and Facebook Creative Testing
(no direct link to watch/listen)
(direct link to watch/listen)
Type:
Webinar
Publication date:
August 13, 2020
Added to the Vault on:
August 23, 2020
Invite a guest
Unlock 600+ mobile growth insights 💎 - for FREE
💎 #
1

Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it

02:40
💎 #
2

It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.

04:12
💎 #
3

Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.

04:48
💎 #
4

CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)

05:41
💎 #
5

IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).

06;40
💎 #
6

Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

08:08
💎 #
7

For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.

09:20
💎 #
8

If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.

10:20
💎 #
9

Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.

12:06
💎 #
10

Testing creatives for ROAS in South Africa is a good proxy for some titles.

13:05
💎 #
11

 Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase

13:45
💎 #
12

ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.

14:50
The gems from this resource are only available to premium members.
  • Unlock access to gems from 170+ mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Receive the hottest gems 🔥 (most valuable insights) each month
Upgrade Your Plan
💎 #
1

Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it

02:40
💎 #
2

It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.

04:12
💎 #
3

Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.

04:48
💎 #
4

CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)

05:41
💎 #
5

IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).

06;40
💎 #
6

Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

08:08
💎 #
7

For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.

09:20
💎 #
8

If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.

10:20
💎 #
9

Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.

12:06
💎 #
10

Testing creatives for ROAS in South Africa is a good proxy for some titles.

13:05
💎 #
11

 Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase

13:45
💎 #
12

ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.

14:50
The gems from this resource are only available to premium members.

Gems are the key bite-size insights "mined" from a specific mobile marketing resource, like a webinar, a panel or a podcast.
They allow you to save time by grasping the most important information in a couple of minutes, and also each include the timestamp from the source.

Become a member to:
  • Unlock access to gems from 170+ mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Receive the hottest gems 🔥 (most valuable insights) each month
GET Access
💎 #
1

Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it

02:40
💎 #
2

It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.

04:12
💎 #
3

Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.

04:48
💎 #
4

CA's new creative testing process:
1. IPM test with no control video and MAI
2. Initial ROAS test with no control video and AEO/VO
3. ROAS scale test with CBO (AEO/VO)

05:41
💎 #
5

IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).

06;40
💎 #
6

Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

08:08
💎 #
7

For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.

09:20
💎 #
8

If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.

10:20
💎 #
9

Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.

12:06
💎 #
10

Testing creatives for ROAS in South Africa is a good proxy for some titles.

13:05
💎 #
11

 Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):
- A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
- The control wins directly but without a challenger being able to get out of the learning phase

13:45
💎 #
12

ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.

14:50
The access to discussions on each resource is only available to premium members.

Growth Gems members discuss resources and their key insights (as well as other mobile growth topics) in the community. It's the perfect way to dig deeper, ask questions and get additional perspectives.
Upgrade to premium to:
  • Unlock access to key insights from 170+ mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the member-only community.
Upgrade Your Plan
The access to discussions on each resource is only available to premium members.

Growth Gems members discuss resources and their key insights (as well as other mobile growth topics) in the community. It's the perfect way to dig deeper, ask questions and get additional perspectives.
Become a member to:
  • Unlock access to gems from 170+ mobile growth resources
  • Define your preferred categories and receive new relevant gems directly in your inbox
  • Discuss key insights (and any other mobile growth topic) in the member-only community.
GET Access

You need to be logged in the community to be able to see the discussion below.
You can also head over directly to this topic in the community

Notes for this resource are currently being transferred and will be available soon.

Why is the control difficult/impossible to beat?

Consumer Acquisition tested 60 different ads for a client and were expecting at least 2/3 winners, but none were able to beat the control.

This got them to duplicate the control several times and test all of the clones. One of the clones actually got an increased CPM.

The dog is replacing the client's ad for confidentiality reasons

[💎 @02:40] Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it.


[💎 @04:12] It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.


[💎 @04:48] Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.


The new creative testing process

Based on these learnings, Consumer Acquisition changed their creative testing process.

[💎 @05:41] CA's new creative testing process:

1. IPM test with no control video and MAI

2. Initial ROAS test with no control video and AEO/VO

3. ROAS scale test with CBO (AEO/VO)



1. IPM Test

[💎 @06:40] IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).


[💎 @08:08] Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

[💎 @09:20] For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.


[💎 @10:20] If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.



2. Initial ROAS Test

[💎 @12:06] Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.


[💎 @13:05] Testing creatives for ROAS in South Africa is a good proxy for some titles.


[💎 @13:45] Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):

  • A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
  • The control wins directly but without a challenger being able to get out of the learning phase

3. ROAS Scale Test

[💎 @14:50] ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.



White papers from Consumer Acquisition: https://www.consumeracquisition.com/whitepapers/


The notes from this resource are only available to premium members.
↘ At this point, you know what to do ↙
Upgrade Your Plan

Why is the control difficult/impossible to beat?

Consumer Acquisition tested 60 different ads for a client and were expecting at least 2/3 winners, but none were able to beat the control.

This got them to duplicate the control several times and test all of the clones. One of the clones actually got an increased CPM.

The dog is replacing the client's ad for confidentiality reasons

[💎 @02:40] Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it.


[💎 @04:12] It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.


[💎 @04:48] Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.


The new creative testing process

Based on these learnings, Consumer Acquisition changed their creative testing process.

[💎 @05:41] CA's new creative testing process:

1. IPM test with no control video and MAI

2. Initial ROAS test with no control video and AEO/VO

3. ROAS scale test with CBO (AEO/VO)



1. IPM Test

[💎 @06:40] IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).


[💎 @08:08] Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

[💎 @09:20] For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.


[💎 @10:20] If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.



2. Initial ROAS Test

[💎 @12:06] Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.


[💎 @13:05] Testing creatives for ROAS in South Africa is a good proxy for some titles.


[💎 @13:45] Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):

  • A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
  • The control wins directly but without a challenger being able to get out of the learning phase

3. ROAS Scale Test

[💎 @14:50] ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.



White papers from Consumer Acquisition: https://www.consumeracquisition.com/whitepapers/


The notes from this resource are only available to premium members.

The detailed notes taken for a resource are an easy way to see the gems in context to get a better understanding. They also include any relevant visuals from the source.
↘ At this point, you know what to do ↙
GET Access

Why is the control difficult/impossible to beat?

Consumer Acquisition tested 60 different ads for a client and were expecting at least 2/3 winners, but none were able to beat the control.

This got them to duplicate the control several times and test all of the clones. One of the clones actually got an increased CPM.

The dog is replacing the client's ad for confidentiality reasons

[💎 @02:40] Don't include the control video when you're doing A/B testing. The control maintains its creative history: it is in the optimized phase, past the learning phase. This gives the control an advantage, making it very difficult to outperform it.


[💎 @04:12] It appears that Facebook is able to use AI/Machine Learning to group similar concepts: very similar concepts seem to inherit the initial IPM starting value of the original winning video. On the other hand, visually different ads do not share the history and will have very different initial IPMs.


[💎 @04:48] Ads that have no visual identity or similarities to the control require more time and more spend for Facebook to understand what they are.


The new creative testing process

Based on these learnings, Consumer Acquisition changed their creative testing process.

[💎 @05:41] CA's new creative testing process:

1. IPM test with no control video and MAI

2. Initial ROAS test with no control video and AEO/VO

3. ROAS scale test with CBO (AEO/VO)



1. IPM Test

[💎 @06:40] IPM Test - start with a brand new campaign, only used for testing (ideally it would even be a brand new account), no control video, 1 ad per ad set to force spend distribution, 3-5 creatives, MAI bidding (regardless of monetization).


[💎 @08:08] Testing creatives for IPM in India is a great proxy for the US, and much less expensive. Consumer Acquisition has found a highly correlated statistical relevance in running in India (estimate of 95%), even when it comes to humor and color palettes.

[💎 @09:20] For IPM tests, do not mix placements: either only FB newsfeed, only Facebook Audience Network. The performance per placement is radically different and the conclusions that you draw would also be different.


[💎 @10:20] If you are an ad-driven only game, you stop at the IPM test phase and select 1-3 winners once creatives are out of the learning phase and you have enough LTV data. If you have a mixed model or monetizing with IAP only, you select 1-3 winners to go to phase 2.



2. Initial ROAS Test

[💎 @12:06] Initial ROAS Test - Brand new campaign, no control video, 1 ad set with many ads, AEO or VO bidding.


[💎 @13:05] Testing creatives for ROAS in South Africa is a good proxy for some titles.


[💎 @13:45] Concerns if you immediately test your new creatives vs. your control (i.e. skip Phase 2):

  • A challenger kills the control over a short period of time and with no statistical relevance, but as the challenger matures its performance drops (and you killed your best-performing video)
  • The control wins directly but without a challenger being able to get out of the learning phase

3. ROAS Scale Test

[💎 @14:50] ROAS Scale Test - Use your CBO campaign, still no control video: copy an existing ad set and use only phase 2 winner(s). As you get results roll them ads on solid audiences but DO NOT put the control in the same ad set until challengers are out of the learning phase.



White papers from Consumer Acquisition: https://www.consumeracquisition.com/whitepapers/