Appendix 3: Error Metrics

NOTE: Only some of these Error Metrics can be used as evaluation metrics. The rest are computed metrics (not used for evaluating which model is better than another).

Mean Absolute Error (MAE)

An error measurement between paired observations expressing the same phenomenon. Examples of Y versus X include predicted versus observed comparisons, subsequent time versus initial time, and one measurement technique versus an alternative measurement technique.

Formula:

Formula for Mean Absolute Error (MAE), showing the average of the absolute differences between actual values and forecasted values across all observations.

Interpretation: Lower is better.

Benefits:

    • Easily interpretable.

    • No favoritism towards over- or under-predictions.

    • Shortcomings:

    • Relative size of the error is not obvious as with percentages.

    Mean Absolute Percent Error (MAPE)

    A prediction accuracy measurement of a forecasting method. Also known as mean absolute percentage deviation.

    Formula:

    Formula for Mean Absolute Percentage Error (MAPE), showing the average percentage difference between actual values and forecasted values across all observations.

    Interpretation: Lower is better.

    Benefits:

    • Easily explainable.

    • Scale-independent / expressed as a percentage.

    Shortcomings:

    • Returns undefined values when forecasting for actual values of zero.

    • Favors models that under-forecast due to heavier penalty on forecasts higher than actuals. Forecasts below actuals cannot be worse than 100% Mean Absolute Percent Error (MAPE).

    Mean Absolute Scaled Error (MASE)

    A measure that determines how effective forecasts generated through an algorithm are by comparing the forecast predictions with the output of a naïve forecasting approach.

    Formula:

    Formula for Mean Absolute Scaled Error (MASE), showing MAE divided by the average absolute difference between consecutive actual values, used to scale forecast error relative to a naïve baseline.

    Interpretation: Lower is better

    Benefit: Independent of the data's scale, so it can be used to compare forecasts across data sets with different scales.

    Mean Asymmetric Over Error (MAOE)

    A combination of both Mean Squared Error (MSE) and Mean Absolute Error (MAE) depending on whether the predicted value is over or under the actual value. If the predicted value is over the actual value, then the error metric applied is MSE (squared difference). If the predicted value is under the actual value, then the error metric applied is MAE (absolute difference).

    Formula:

    Formula for Mean Asymmetric Overestimation Error (MAOE), where squared error is used when actual values are less than or equal to forecasts, and absolute error is used when actual values exceed forecasts, averaged across all observations.

    Interpretation: Lower is better.

    Benefit: Favors models that under predict. May be useful if there is a difference in penalty in real world application for over predictions.

    Shortcomings: Can over penalize a model for over predicting just once or twice way more than under predicting consistently.

    Mean Asymmetric Under Error (MAUE)

    Similar to Mean Asymmetric Over Error (MAOE), but applies Mean Squared Error (MSE) to predictions below the actual value and Mean Absolute Error (MAE) to predictions above the actual value.

    Formula:

    Formula for Mean Asymmetric Underestimation Error (MAUE), where absolute error is used when actual values are less than or equal to forecasts, and squared error is used when actual values exceed forecasts, averaged across all observations.

    Interpretation: Lower is better.

    Benefit: Favors models that over predict. May be useful if there is a penalty differences in real world application for under predictions.

    Shortcomings: Can over penalize a model for under predicting just once or twice way more than over predicting consistently.

    Mean Bias Error (MBE)

    Mean Bias Error (MBE) is primarily used to estimate the average bias in a model and determine what is needed to correct the model bias. MBE captures the average bias in the prediction.

    Formula:

    Formula for Mean Bias Error (MBE), showing the average difference between predicted values and observed values across all observations.

    Interpretation: Typically not used as a measure of model error, as high individual errors in prediction can also produce a low MBE. MBE reflects a prediction's average bias. A positive value represents overestimating bias and a negative value represents an underestimating bias.

    Shortcomings: This is typically not used to measure model error.

    Mean Percent Error (MPE)

    The computed average of percentage errors by which a model's forecasts differ from actual values of the forecasted quantity.

    Formula:

    Formula for Mean Percentage Error (MPE) showing the average of percentage errors by which a model’s forecasts differ from actual values across all observations.

    Interpretation: Closer to zero is better.

    Benefits: Can be used as a measure of the bias in the forecasts

    Shortcomings:

    • Measure is undefined when a single actual value is zero.

    • Positive and negative forecast errors can offset each other.

    Mean Squared Error (MSE)

    The average squared difference between the estimated values and the actual value. MSE is a risk function, corresponding to the expected value of the squared error loss.

    Formula for Mean Squared Error (MSE), showing the average of the squared differences between predicted values and actual values across all data points, with definitions for each variable included below.

    Interpretation: Lower is better.

    Benefits:

    • Increased penalty for larger errors.

    • Ensures positives values for easy interpretation.

    Shortcomings:

    • Less intuitive than MAE.

    Mean Squared Logarithmic Error (MSLE)

    A measure of the ratio between the true and predicted values. MSLE is a variation of Mean Squared Error (MSE).

    Formula:

    Formula for Mean Squared Logarithmic Error (MSLE), showing the average of the squared differences between the logarithms of actual values and predicted values across all observations.

    Interpretation: The loss is the mean over the seen data of the squared differences between the log-transformed true and predicted values.

    Benefit: Treats small differences between small true and predicted values approximately the same as large differences between large true and predicted values.

    Shortcomings: Penalizes underestimates more than overestimates.

    Median Absolute Error (MedAE)

    The loss is calculated by taking the median of all absolute differences between the target and the prediction. If ŷ is the predicted value of the sample and y1 is the corresponding true value, then the median absolute error estimated over n samples is defined as follows:

    Formula:

    Formula for Median Absolute Error (MedAE), showing the median of the absolute differences between actual values and predicted values across all observations.

    Interpretation: Lower is better

    Benefit: Robust to outliers

    Symmetric Mean Absolute Percent Error (SMAPE)

    An accuracy measure based on percentage (or relative) errors.

    Formula:

    Formula for Symmetric Mean Absolute Percentage Error (SMAPE), showing the average percentage difference between actual and forecast values normalized by their average, with definitions for actual value (A_t) and forecast value (F_t) included.

    The absolute difference between At and Ft is divided by half the sum of absolute values of the actual value At and the forecast value Ft. The value of this calculation is summed for every fitted point t and divided again by the number of fitted points n.

    Interpretation: Lower is better

    Benefits:

    • Expressed as a percentage.

    • Lower and upper bounds (0% - 200%).

    Shortcomings:

    • Unstable when both the true value and the forecast value are close to zero.

    • Not as intuitive as MAPE (Mean Absolute Percent Error).

    • Can return a negative value.

    R2 (R-Squared)

    A statistical measure of fit that indicates how much variation of a dependent variable is explained by the independent variables in a regression model.

    Formula:

    Worked example showing calculation of the correlation coefficient (R²), including a table of X and Y values with their squares and products, followed by the formula substitution and steps leading to a final result of R² = 1, indicating perfect correlation.

    Interpretation: Higher is usually better; Measures the strength of the relationship between independent and dependent variables in a regression model; Ranges between 0 (No Correlation 0 and 1 (Perfect Correlation)

    Benefits: Commonly used as a measurement technique

    Shortcomings:

    • Sometimes, a high R-Squared value can indicate problems with the regression model.

    • Does not reveal the causation relationship between the independent and dependent variables.

    Symmetric Mean Absolute Percent Error (SMAPE)

    A measure to compare true observed response with predicted response in regression tasks.

    Formula:

    Formula for Symmetric Mean Absolute Percentage Error (SMAPE), showing the average percentage difference between actual values and forecast values normalized by their average, with definitions for actual value (A_t) and forecast value (F_t) included.

    Interpretation: The value of this calculation is summed for every fitted point t and divided again by the number of fitted points n.

    Benefits: Has both a lower bound and an upper bound.

    Shortcomings: If the actual value or forecast value is 0, the value of error goes to the error upper limit.