Data Science Series- Uplift Modeling: measuring true campaign impact

You’ve just finished running a new advertising campaign. Now you want to know the answer to your obvious and most pressing question: did it work? That is, did the campaign create uplift, causing more customers to purchase your product than would have without the campaign?

Naïve and incorrect approach

The conversion rate after receiving the offer was 7%, and so you conclude that 7% of customers bought your product because they saw the ad.

This is incorrect. What if those same customers were going to purchase your product anyways? Without comparing the conversion rate of those who saw the ad to those that didn’t, you aren’t getting a true measure of how much ‘lift’ your campaign actually caused.

In other words, you should not use the approach outlined above.

Basic solution: controlled testing

It is always a good idea to send your campaign to some potential customers (your ‘test’ group) and not send it to others (your ‘control’ group). These two groups should be as comparable as possible in all other ways.

For example, you should not send your campaign to women only and not to men and then try to compare the conversion rate between these groups; you won’t be able to separate out whether the impact was due to gender or to the campaign.
Once you have run your test on comparable results, you can analyze the results and measure the uplift that the campaign created. As an extremely basic example, if the audience segment that was sent the campaign had a conversion rate of 7%, but the segment who were not sent the offer in the same time period had a conversion rate of 5%, then the true uplift of your campaign is 2%. Although this may paint a bleaker picture than using the naïve and incorrect approach,  it is the true measure of lift for your campaign rather than assuming causality and not comparing what would have happened without your campaign.

That being said, it is always a good idea to check your results for statistical significance, to ensure you have found true uplift and that the difference between the groups is likely not due to chance.

Advanced solution: customized model for efficient targeting

Once you have measured the uplift of your campaign, you might be curious to keep going and dig even deeper. Who did the campaign have the greatest impact on? Did it increase conversion for certain demographics more so than others? Were there any customers who didn’t convert because of the campaign? Is there any way to use the results of this campaign to decide who on your customer list should be targeted in the future?

If you possess user level data for both customers who were targeted by an ad campaign, and for those who were not, a customized uplift model is a great way to get user-level insights from your data. As with controlled testing, you will want the customers who were sent the ad and those who not to be as comparable as possible in all other ways. Then, a model can be built using machine learning and statistical techniques to predict how likely it was that someone would have purchased with and without having seen the ad.

You will gain two types  of insights from this model:

1. The uplift the campaign caused for each customer within your customer list. With this data also comes the cutoffs for which customers were in the top 25% of uplift, or those who saw more than 10% uplift as a result of the campaign.
2. Profile of demographics and traits associated with higher or lower uplift. This is based on the variables that were input into the model, so if you knew the gender, region, and urban vs. suburban status of your customers, this would predict which combinations of these resulted in the highest uplift.
   

Once you have obtained this information, you will be able to spend your campaign budget more efficiently. For example, you could increase ROI by targeting customers with high uplift, and decrease costs by no longer displaying ads to customers with low or negative uplift. You will even be able to acquire new customers by seeking demographic groups that match your demographic profiling.

This may also bring up more questions for you: Why didn’t a certain demographic respond well to your campaign? Can we more effectively reach that demographic through different forms of advertising? This is where market research questions will come in, and you you will have to do further research in order to get the answers to those questions.

Conclusion

Uplift modelling can help decrease unnecessary and potentially harmful costs, increase ROI through efficient targeting, and grow your user base through demographic profiling of those most responsive to advertising.