Choosing the right error metric for your predictive model

In this post, a member of the Cardinal Path’s Data Science team, Danika Law, explains why selecting the right error metric for your business problem is so crucial. In the early project phases of running a dashboard or predictive model, we always address the client’s needs first and foremost. This post explains why, to a data scientist, selecting the right error metric is  a necessary step to ensure accuracy of the model and solve for the problem at hand.

When you are building a predictive model, no matter what problem you are trying to solve for, you should always specify your error metric right from the start of the forecasting project, and make sure that it is in line with your business goals. Some error metrics punish overestimates more than underestimates, while others punish larger errors exponentially. Having the errors punished according to what most makes sense for your business will lead to a better model.

There should be one main consideration when choosing your error metric: what is the cost associated with making an incorrect prediction, and where is this cost the largest?

• Do we want to punish overestimates or underestimates more heavily?
  • For example, when classifying churners (those customers who are going to drop off), if the cost of misclassifying other customers as potential churners is less than the cost of missing some actual churners, we will want to punish underestimates more heavily. This helps ensure we don’t miss any churners.
• Are there segments which have greater costs associated with incorrect predictions?
  • For example, let’s say we are classifying customers into “High Value Customer”, “Medium Value” and “Low Value” based on the predicted lifetime revenue they will generate for the business. If we misclassify someone as a “High Value Customer” and then spend a lot of advertising resources on them and end up not being a high value customer, the return on investment will be low. It is for this reason that the value needs to be weighted accordingly.
• Should we punish larger errors or smaller errors more heavily?
  • For example, let’s say we are forecasting revenue by marketing channel. The predictions will be used to help plan what to expect, so having very high or very low projected revenue that is incorrect could lead to making poor business decisions. On the other hand, being off by a small amount here and there doesn’t change the decisions made to such a large degree. In cases like this, we’d want to punish large errors more heavily than small errors.

Once you have specified how you want your error metric to punish errors, you’ll need to select one that fits your business needs. Here are some resources that will help you select:

• Kaggle
• Machine Learning Mastery

Ensuring that you specify your business problem correctly, and choose the correct error metric will help ensure that the cost of making a mistake is balanced out by how you evaluate your model.