Accuracy Degradation Over Time

It is normal and expected that model accuracy degrades over time. This is a symptom of the underlying data patterns changing since the model was initially deployed.

The Importance of Model Refits

Model Refit: Taking a deployed model and refitting the same model configuration on the latest data.

Sensible Machine Learning attempts to automatically protect against model accuracy degradation by giving the option to Refit with Latest Data before running its next prediction. This functionality defaults to True, and the prediction takes longer because every model is refit. This functionality refits the production model with the latest refreshed source data. Often, this functionality is enough to keep a model healthy. In some cases, this may even increase model accuracy over time leading to a positive health score.

OneStream recommends leaving Refit with Latest Data set to True unless the project needs quick and frequent predictions.

The Importance of Model Rebuilds

Rebuild: A rebuild consists of creating new models, data sources, and configurations for a set of targets.

The underlying data patterns are expected to change over the course of time which will cause model accuracy degradation. This implies that:

  • Some of the features that were once important may no longer be useful.

  • There may be new events that can positively influence accuracy.

  • A different intelligent model may better represent the data.

The Purpose of a Partial Rebuild

In large projects, there may be some targets that degrade to unacceptable levels faster than others. Instead of rebuilding all targets and spending time retraining, the partial rebuild option allows you to rebuild only the poorly performing targets.

Know When to Rebuild

Sensible Machine Learning provides indicators throughout the Model Utilization section.

Model Health Distribution Chart: The chart is available on the Manage Health page and the Analysis Overview page. It plots and color codes the health score for all targets; green = healthy, yellow = warning, and red = unhealthy. If a cluster of red is found with mostly green, this may be an indicator to execute a Partial Rebuild to get the red targets back into the green. If there is mostly yellow and red, this may be an indicator to execute a Full Rebuild.

XperiFlow Suggestions: A message grid view that lets you know if a rebuild is suggested.