The Art & Science of Proper Attribution & Incrementality

Item not bookmarked
Resource bookmarked
Bookmarking...
⛏️
Guest Miner:
Sylvain Gauchet
Review star
Review star
Review star
Review star
Review star
💎  x
6

Thomas Hopkins (Head of Performance & Lifecycle Marketing at Masterclass - online learning) talks about incrementality testing, differences between CPA and CPIA, several ways to assess ROAS so you can assess the profitability of your channels and match-market testing.

Source:
The Art & Science of Proper Attribution & Incrementality
(no direct link to watch/listen)
(direct link to watch/listen)
Type:
Presentation
Publication date:
February 25, 2020
Added to the Vault on:
March 24, 2020
Invite a guest
These insights were shared through the free Growth Gems newsletter.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
💎 #
1

You have to look at total profit or total projected profit to be able to see if it is going up or down. If it is going down, your CPIA (Cost Per Incremental Acquisition) is most likely above your LTV. 

05:12
💎 #
2

Lift tests are really hard to do because the hold out group is seeing ads until you start the hold out. But they have already seen your ads! So you need to run the lift test for a long time. 

08:55
💎 #
3

Use a "How did you hear about us survey" as another way to calculate ROAS per channel. 

13:33
💎 #
4

For match-market testing take 2 geos (or group of geos) that track/behave the same. Instead of splitting the audience in 2 groups for an A/B test, use one group of geos as A and the other group of geos as B. This allows you to test multiple channels at the same time in group A and use group B as control. 

16:58
💎 #
5

For match-market testing, start by choosing cities that do not represent too much spend so you can get approval (5% to 10% of your sales should be the maximum). Turn off spend then gradually add your channels and increase spend. 

18:03
💎 #
6

Your CPIA curve keeps going up more and more and you want to stop spending when it reaches your LTV. Easiest way is to constantly be monitoring your projected profit (LTV - spend). 

20:20
The gems from this resource are only available to premium members.
💎 #
1

You have to look at total profit or total projected profit to be able to see if it is going up or down. If it is going down, your CPIA (Cost Per Incremental Acquisition) is most likely above your LTV. 

05:12
💎 #
2

Lift tests are really hard to do because the hold out group is seeing ads until you start the hold out. But they have already seen your ads! So you need to run the lift test for a long time. 

08:55
💎 #
3

Use a "How did you hear about us survey" as another way to calculate ROAS per channel. 

13:33
💎 #
4

For match-market testing take 2 geos (or group of geos) that track/behave the same. Instead of splitting the audience in 2 groups for an A/B test, use one group of geos as A and the other group of geos as B. This allows you to test multiple channels at the same time in group A and use group B as control. 

16:58
💎 #
5

For match-market testing, start by choosing cities that do not represent too much spend so you can get approval (5% to 10% of your sales should be the maximum). Turn off spend then gradually add your channels and increase spend. 

18:03
💎 #
6

Your CPIA curve keeps going up more and more and you want to stop spending when it reaches your LTV. Easiest way is to constantly be monitoring your projected profit (LTV - spend). 

20:20
The gems from this resource are only available to premium members.

Gems are the key bite-size insights "mined" from a specific mobile marketing resource, like a webinar, a panel or a podcast.
They allow you to save time by grasping the most important information in a couple of minutes, and also each include the timestamp from the source.

💎 #
1

You have to look at total profit or total projected profit to be able to see if it is going up or down. If it is going down, your CPIA (Cost Per Incremental Acquisition) is most likely above your LTV. 

05:12
💎 #
2

Lift tests are really hard to do because the hold out group is seeing ads until you start the hold out. But they have already seen your ads! So you need to run the lift test for a long time. 

08:55
💎 #
3

Use a "How did you hear about us survey" as another way to calculate ROAS per channel. 

13:33
💎 #
4

For match-market testing take 2 geos (or group of geos) that track/behave the same. Instead of splitting the audience in 2 groups for an A/B test, use one group of geos as A and the other group of geos as B. This allows you to test multiple channels at the same time in group A and use group B as control. 

16:58
💎 #
5

For match-market testing, start by choosing cities that do not represent too much spend so you can get approval (5% to 10% of your sales should be the maximum). Turn off spend then gradually add your channels and increase spend. 

18:03
💎 #
6

Your CPIA curve keeps going up more and more and you want to stop spending when it reaches your LTV. Easiest way is to constantly be monitoring your projected profit (LTV - spend). 

20:20

Notes for this resource are currently being transferred and will be available soon.

CPA vs. CPIA

If you look only at your CPA, you see it going up but because it is under your $250 target you keep spending. But each new acquired user costs more.


[💎@05:12] You have to look at total profit or total projected profit to be able to see if it is going up or down. If it is going down, your CPIA (Cost Per Incremental Acquisition) is most likely above your LTV.


Google and Facebook attribution


Platform reported sum of purchases is usually more than the actual purchases, because channels are overlapping.

It says YouTube because that's where Masterclass spends the most on when it comes to Google.

Facebook is great at cross-device attribution so often over attributes.


[💎@08:55] Lift tests are really hard to test because the hold out group is seeing ads until you start the hold out. But they have already seen your ads! So you need to run the lift test for a long time.


On YouTube, there are examples of seeing 50/60% ROAS when it should be 150% ROAS.

  • One factor is if you are not logged in YouTube
  • Other is that search often takes credit


Incrementality


Regression Analysis

The dependent variable is the revenue (or projected revenue) based on your LTV. All the other variables are based on your channel spend.


How did you hear about us survey

The line charts shows the ROAS.

[💎@13:33] Use a "How did you hear about us survey" as another way to calculate ROAS per channel.

Based on the response % per channel, take that response value and take the cost of the spend in that channel to figure out what the ROAS is.

Example: Purchasers from Facebook * Perceived LTV for customer / Cost.


Looking at everything together

Doing this kind of analysis will show you some clear trends and allow to identify where you have unknowns.


Match-Market testing

Match-market testing is designed to define what the best mix of channels is. You can find the recommended Google white paper on match-market testing here.


You can look at 4 channels just for FB/Google:

  1. Facebook prospecting
  2. Facebook retargeting
  3. Google prospecting
  4. Google retargeting


[💎@16:58] For match-market testing take 2 geos (or group of geos) that track/behave the same. Instead of splitting the audience in 2 groups for an A/B test, use one group of geos as A and the other group of geos as B. This allows you to test multiple channels at the same time in group A and use group B as control.


  1. Work with data science team to find out which geos (cities, countries) you should use)
  2. Set up your tests based on which channels you want to tests


[💎@18:03] For match-market testing, start by choosing cities that do not represent too much spend so you can get approval (5% to 10% of your sales should be the maximum). Turn off spend then gradually add your channels and increase spend.


[💎@20:20] Your CPIA curve keeps going up more and more and you want to stop spending when it reaches your LTV. Easiest way is to constantly be monitoring your projected profit (LTV - spend).


The notes from this resource are only available to premium members.

CPA vs. CPIA

If you look only at your CPA, you see it going up but because it is under your $250 target you keep spending. But each new acquired user costs more.


[💎@05:12] You have to look at total profit or total projected profit to be able to see if it is going up or down. If it is going down, your CPIA (Cost Per Incremental Acquisition) is most likely above your LTV.


Google and Facebook attribution


Platform reported sum of purchases is usually more than the actual purchases, because channels are overlapping.

It says YouTube because that's where Masterclass spends the most on when it comes to Google.

Facebook is great at cross-device attribution so often over attributes.


[💎@08:55] Lift tests are really hard to test because the hold out group is seeing ads until you start the hold out. But they have already seen your ads! So you need to run the lift test for a long time.


On YouTube, there are examples of seeing 50/60% ROAS when it should be 150% ROAS.

  • One factor is if you are not logged in YouTube
  • Other is that search often takes credit


Incrementality


Regression Analysis

The dependent variable is the revenue (or projected revenue) based on your LTV. All the other variables are based on your channel spend.


How did you hear about us survey

The line charts shows the ROAS.

[💎@13:33] Use a "How did you hear about us survey" as another way to calculate ROAS per channel.

Based on the response % per channel, take that response value and take the cost of the spend in that channel to figure out what the ROAS is.

Example: Purchasers from Facebook * Perceived LTV for customer / Cost.


Looking at everything together

Doing this kind of analysis will show you some clear trends and allow to identify where you have unknowns.


Match-Market testing

Match-market testing is designed to define what the best mix of channels is. You can find the recommended Google white paper on match-market testing here.


You can look at 4 channels just for FB/Google:

  1. Facebook prospecting
  2. Facebook retargeting
  3. Google prospecting
  4. Google retargeting


[💎@16:58] For match-market testing take 2 geos (or group of geos) that track/behave the same. Instead of splitting the audience in 2 groups for an A/B test, use one group of geos as A and the other group of geos as B. This allows you to test multiple channels at the same time in group A and use group B as control.


  1. Work with data science team to find out which geos (cities, countries) you should use)
  2. Set up your tests based on which channels you want to tests


[💎@18:03] For match-market testing, start by choosing cities that do not represent too much spend so you can get approval (5% to 10% of your sales should be the maximum). Turn off spend then gradually add your channels and increase spend.


[💎@20:20] Your CPIA curve keeps going up more and more and you want to stop spending when it reaches your LTV. Easiest way is to constantly be monitoring your projected profit (LTV - spend).


The notes from this resource are only available to premium members.

CPA vs. CPIA

If you look only at your CPA, you see it going up but because it is under your $250 target you keep spending. But each new acquired user costs more.


[💎@05:12] You have to look at total profit or total projected profit to be able to see if it is going up or down. If it is going down, your CPIA (Cost Per Incremental Acquisition) is most likely above your LTV.


Google and Facebook attribution


Platform reported sum of purchases is usually more than the actual purchases, because channels are overlapping.

It says YouTube because that's where Masterclass spends the most on when it comes to Google.

Facebook is great at cross-device attribution so often over attributes.


[💎@08:55] Lift tests are really hard to test because the hold out group is seeing ads until you start the hold out. But they have already seen your ads! So you need to run the lift test for a long time.


On YouTube, there are examples of seeing 50/60% ROAS when it should be 150% ROAS.

  • One factor is if you are not logged in YouTube
  • Other is that search often takes credit


Incrementality


Regression Analysis

The dependent variable is the revenue (or projected revenue) based on your LTV. All the other variables are based on your channel spend.


How did you hear about us survey

The line charts shows the ROAS.

[💎@13:33] Use a "How did you hear about us survey" as another way to calculate ROAS per channel.

Based on the response % per channel, take that response value and take the cost of the spend in that channel to figure out what the ROAS is.

Example: Purchasers from Facebook * Perceived LTV for customer / Cost.


Looking at everything together

Doing this kind of analysis will show you some clear trends and allow to identify where you have unknowns.


Match-Market testing

Match-market testing is designed to define what the best mix of channels is. You can find the recommended Google white paper on match-market testing here.


You can look at 4 channels just for FB/Google:

  1. Facebook prospecting
  2. Facebook retargeting
  3. Google prospecting
  4. Google retargeting


[💎@16:58] For match-market testing take 2 geos (or group of geos) that track/behave the same. Instead of splitting the audience in 2 groups for an A/B test, use one group of geos as A and the other group of geos as B. This allows you to test multiple channels at the same time in group A and use group B as control.


  1. Work with data science team to find out which geos (cities, countries) you should use)
  2. Set up your tests based on which channels you want to tests


[💎@18:03] For match-market testing, start by choosing cities that do not represent too much spend so you can get approval (5% to 10% of your sales should be the maximum). Turn off spend then gradually add your channels and increase spend.


[💎@20:20] Your CPIA curve keeps going up more and more and you want to stop spending when it reaches your LTV. Easiest way is to constantly be monitoring your projected profit (LTV - spend).