This page includes Forecast, Feature Impact, Beeswarm, Tug of War, Waterfall, and Periodic Explanation pages. Use the Forecast page in the Analysis section to analyze results from model forecasts, different builds, and different stages of the project. This page visualizes the forecasted values for each model and back filled actuals for each target. It is similar to the Arena Summary view of the Model Build phase Pipeline section, but this page displays in-production results.
This page includes an Accuracy view (default), an Impact view, and an Explanation view. Use the fields at the top of the page to filter the information displayed on any of the views. These fields include:
NOTE: Feature impact data is dependent on the type of model. Not all models have feature impact data.
Analyze Forecast Accuracy View Information
The Forecast view allows the user to view source actuals overlaid with the selected model’s predictions. Error metrics are displayed in the correlated subplot below. Select a datapoint on the plot to view explanations for a given prediction..
This helps to answer questions such as:
-
Which type of model has the best accuracy for a given target?
-
By how much did the model win?
It also helps you understand how closely the forecasted values overlay the actuals in the line chart, which can provide answers to questions such as:
-
Are there spikes that aren’t being caught by the forecasts?
-
If so, could adding any events help catch these spikes?
The table on this page displays:
-
The model algorithms run for a given target (such as XGBoost, CatBoost, or Shift).
-
The type of model algorithm (ML, Statistical, or Baseline).
-
The evaluation metric (such as Mean Squared Error, Mean Absolute Error, and Mean Absolute Percentage Error) and the associated score.
-
The train time (How long did it take to train the given model during Pipeline?).
With all the configurations and the newly found important features, the engine runs multiple models per configuration to find the best. This process involves hyperparameter tuning each model on multiple splits of the data and then saving the accuracy metrics of each model.
The line chart corresponds to the highlighted model in the table. It visualizes both the predictions made for the historical actual test period and the historical actuals. The time frame in this chart is only a subset of the total time frame for the historical data, as this time frame is for a specific portion of a split.
At a high level, this page provides the view of how the best version of each model (such as XGBoost, CatBoost, ExponentialSmoothing, Shift, and Mean) has performed against unseen historical data for each target and more specifically, on the test set of the historical data.
The optimal error metric score can be the lowest score, the highest score, or the closest to zero. It is dependent on the type of metric. See Appendix 3: Error Metrics for more information about the error metrics SensibleAI Forecast uses.
If you select a prediction point on the line chart, it will open a graph below providing prediction explanations for whichever data point was selected. This will include feature explanations along with the impact contribution of the feature when selected.
Analyze Feature Impact
The Feature Impact view shows the hierarchical (Grouped) or individual (Detail) feature impact percentages aggregated across all the model’s prediction values. The user can click on a feature name from the table to view its actual values compared with the predictions and actuals in the visual on the right. The prediction and actual values are bound by the primary y-axis on the left hand side of the graph, while the feature values are bound by the secondary y-axis on the right. The feature impact score shows how much influence the feature had for a given model.
NOTE: Feature impact data is dependent on the type of model. Not all models have feature impact data.
Analyze Beeswarm View
The Beeswarm view visualizes the impact of features on model predictions by displaying SHAP values as individual dots, where each dot represents a feature’s SHAP value for a specific instance. Features are ordered by importance, with the most influential ones at the top. The color of each dot indicates the actual feature value. The horizontal spread of dots reflects the distribution of SHAP values for each feature, showing how much they contribute to model predictions. Plot is based on source granularity and changing the Analysis Granularity will not change the plot.
Analyze Tug of War View
The Tug of War view visualizes the four most significant positive and negative prediction explanations for all dates at the selected Analysis Granularity of the selected model. Compare with base predictions and actual prediction values for all dates. Users can toggle any component of the visual on or off by selecting it from the legend on the right. Users can also narrow in on specific dates by interacting with the range slider. By clicking on one of the bars in the graph, it will expand the feature contributions.
Analyze Prediction Explanations Waterfall View
View the aggregated predictions explanations for the selected forecast version of the selected model. Feature groupings are bucketed based on the Feature Grouping selection. Hover over a grouping to see the specific features that make up the given group. The figure below is only based on the source granularity. Only the Total Data Points will change when Analysis Granularity is changed.
Analyze Periodic Explanations View
View the aggregated predictions explanations for the selected forecast version of the selected model. Feature groupings are bucketed based on the Feature Grouping selection. Click on a grouping to see the specific features that make up the given group. Click on a column header to generate a waterfall explanation chart for the given date range. The figure below is only based on the source granularity. Only the Total Data Points will change when Analysis Granularity is changed.
