Use Historical and Period-over-Period Alerts to Reduce False Positives From Cyclical Changes
During an alert review, operations engineer Lin is about to raise the CPU threshold for a batch-processing host. An alert fires every day after the job starts. After checking job progress and business response times, the on-call team usually confirms that it is another expected peak.
But when he expands the chart to include quieter hours, he finds a different change: CPU utilization is noticeably higher than usual, yet never reaches the existing alert threshold. Raising that threshold further would make these changes even harder for the same rule to detect.
Lin needs to decide both when a response is warranted and how to reduce repeated checks. This simplified policy review shows how historical comparisons can help him make that decision. All values are examples, not universal thresholds or measured outcomes.
Historical and period-over-period alerts turn past behavior into an explicit reference, so cyclical peaks do not need to be explained from scratch every time, while deviations during quieter periods are more likely to receive attention.
