“When a measure becomes a target, it ceases to be a good measure.” – Goodhart’s Law
Goodhart’s Law reminds us that oftentimes, setting a target (such as a KPI), can change the way in which people work toward the goals these targets are meant to help us reach.
An absurd example of this occurred in India during the time of the British rule. There was an abundance of venomous cobra snakes loose on the streets, so the British government offered money for each dead cobra that was turned in. Rather than resulting in everyone catching many cobras and handing them in to collect their bounty, thus reducing the cobra population, people actually began to breed cobras and hand them over to the government for the bounty. This was obviously not a solution to the problem. To add to the problem, once the government caught wind of these breeders, they stopped paying the bounty, so the breeders released all of the cobras– resulting in an even higher population of cobras.
How does Goodhart’s law relate to attribution models?
So… (now that you’ve got your legs safely tucked up beneath yourself on your chair), let’s shift our focus away from snakes and take a look at why Goodhart’s Law not only applies to examples like this, but how it can apply to the way in which we look at incentives when measuring success for any goal. Take attribution, where we try to determine which marketing channels have the largest impact on conversion, for example. How does an attribution model change the incentives of marketers?
Ideally, it would incentivize marketers to increase the ROI of the channel that they manage, and incentivize them to drive more revenue. For example, channels shown to have low attribution values would aim to increase their value through optimization efforts such as cutting spend in low-performing geographies, carrying out more precise targeting, and so on. Channels shown to have high attribution values would find out if increased budget would lead to revenue at a similar rate, or if there is a point where they would experience diminishing returns to the investment in that channel.
However, for some attribution models, in particular for last touch attribution, this may not be the case. Last touch attribution provides all of the value for the marketing channel that immediately precedes the conversion, ignoring the impact that any other channels may have seen on conversion. This often leads to undervaluation of channels typically earlier in the conversion path, such as display, and an overvaluation of channels typically at the end of the conversion path, such as paid search and branded organic search.
The problem with last-touch
Furthermore, last-touch attribution may change the incentives for your marketers.
Rather than giving credit to marketers for contributing to increased ROI, they would get credit for being the last in the conversion path. This means that theoretically they could ‘game’ the system by blasting ads at people who have already shown interest in buying your product, hoping to be the last one to touch them before they convert.
This would lead to mostly remarketing campaigns and branded search: displaying ads to people who are already interested in your product (visited the website) and are already likely to convert. It essentially becomes a race to be the last person to touch that user.
This is another reason why we need a data-driven attribution model. Rather than last-touch, or some other rules-based attribution model, a data-driven attribution model that is shown to lead to more efficient spending and increased ROI is what is really needed.
How to avoid the last-touch trap
So, I encourage you to take a step beyond rules-based attribution models like last-touch attribution. But while doing so, ensure that the data-driven attribution model doesn’t steer the incentives away from your true business goal. Is the model biased to traffic in different time periods, like at the start or end of the month, leading to incentives to serve at those times rather than at the times that generally lead to increased ROI? Does it continue to undervalue upper-funnel tactics like display, leading to focusing purely on lower-funnel tactics like paid search and to an unhealthy conversion funnel overall?
Most importantly, after implementing your attribution model, and (here’s the important part) acting upon the recommendations from it, do you see increased overall revenue or ROI to your overall spend? If your model is working, and creating good incentives and recommendations, you should be seeing these results. Not only should credit be shifted between channels, but shifting the budgets to where the model claims it will have the highest impact should lead to increased revenue, if the model is working.
Goodhart’s Law, stating that when a measure becomes a target it ceases to be a good measure, is just another reason why we need to start attributing credit among media channels in different ways, rather than last touch attribution. Good options for this are data driven attribution or media mix modelling. Beyond that, it’s an important reminder to think about how changing KPIs will change the incentives. We need to ensure that those incentives remain in line with your actual business goals, regardless of the analysis being done.
