The monitor system sits between the prediction pipeline and downstream consumption. It determines which single model should be used for each target going forward. This chosen model is called the Effective Model. It is the model whose forecast appears in your consumption groups, dashboards, and exports.
Solution Utilities
-
Global Exclusion Rule: A global exclusion rule permanently removes certain models from the candidate pool before any rules run. It applies to all targets in your project.
-
Monitor Rule: A monitor rule is a user-defined configuration that controls effective model selection for a set of targets. Each rule has five components: Name, Severity, Scope, Flag, and Effective Criteria.
Create Global Exclusion Rule
Global exclusion rules can be converted to eliminate models for all targets in your project.
To create a Global Exclusion Rule, complete the following steps:
-
Enable the global exclusion toggle.
-
Configure the condition through the input boxes.
-
Save exclusion.
Create Monitor Rule
A monitor rule has five main components:
-
Name: A human readable label that uniquely identifies the rule within your project
-
Severity: An informational label stamped on the results. Severity does not affect which model is selected. It is used for visual indicators in the prediction summary dashboard so you can quickly spot targets that warrant attention.
-
Scope: The scope defines which targets the rule claims and ensures an effective model selection based on the conditions configured in the rule.
-
Flag: A flag identifies undesirable models. A model that satisfies a flag condition is “flagged”. Flag evaluations are always recorded for every model in the rule’s scope, regardless of whether the model is ultimately selected.
-
Effective Criteria: The effective criteria defines the positive selection filter for the effective model. Only the models that satisfy the effective criteria conditions are eligible to be chosen.
-
Priority: The priority field in the effective criteria is an integer that controls execution order relative to other monitor rules. Rules with higher priority values will run first. Rules with no effective criteria will run last. Priorities must be unique across all rules.
-
Include Flag in EC: This is a boolean toggle that, when enabled, creates a combined filter that ensures that the selected effective model satisfies the effective criteria conditions AND is not flagged by the flag condition.
To create a Monitor Rule, complete the following steps:
-
Create a unique name for the monitor rule
-
Select the scope of the rule from the drop-down menu or select All Remaining.
-
Select a severity level for your flag condition: Critical, Warning, or Info.
-
Configure the flag condition with the drop-downs.
-
Toggle the effective criteria option if your rule needs specific conditions for the effective model selection.
-
Select evaluation priority value for the rule.
-
Set your effective criteria conditions.
-
Toggle if the effective model selection should also pass the flag condition.
-
-
Save the Rule.
After setting your monitor rules, these rules will select the effective model for every subsequent prediction ran in the specific project. If application of these monitor rules is necessary for previously ran predictions, navigate to the Run On Previous Forecast Versions tab, review the monitor rules, select a desired forecast version, and select Run Monitor.
Analyze Monitor Results for Targets
Use the prediction summary dashboard to analyze which model was selected for each specific target. The prediction summary dashboard can be found under Utilization > Analysis > Forecast.
Prediction Summary Model Arena
When you select a target, the Model Arena shows all models that competed for that target. The arena displays:
-
Model name and rank: ranked by the selected evaluation metric.
-
Effective Model Indicator: the effective model row is displayed in bold with “(Effective)” appended to the model name.
-
Monitor Status Column: shows flag evaluation status for each model at the given forecast version.
-
Red flag icon: the model was flagged by a rule with critical severity.
-
Yellow flag icon: the model was flagged by a rule with warning severity.
-
Blue info icon: the model was flagged by a rule with info severity.
-
Green check icon: the model was evaluated by rules but passed all flag conditions.
-
Dash icon: indicates that these are historical rows that did not get processed by the monitor system.
Prediction Summary Targets Table
Monitor Status Column: Shows flag evaluation status for that target at the latest forecast version. It uses the same icon vocabulary as the Model Arena table.
Hovering over the icon reveals a tooltip with the detailed flag evaluation comments for that target, showing each condition that was evaluated and whether it tripped.
Effective Model Lineage
The effective model lineage window is a chronological timeline of every selection event and override for the current target. It will show:
-
Which model was selected at each forecast version.
-
The selection source: Data Monitor, Rank Fallback, and User Override.
-
Whether the model was flagged.
-
Any override comments.
-
Timestamp of each event.
If the effective model selection is inadequate; the user can manually override the selection through the effective model lineage window. Overrides take precedence over the automated monitor selection. Overrides are per-target, per-forecast-version. Overriding the effective model for one forecast version does not affect other versions. If you run a new prediction, the new forecast version will have its own automated selection.
