Merge pull request #347 from grafana/ruler-alerts

Replace ruler alerts, and add playbooks.

Author: pracucci

CHANGELOG.md

@@ -17,6 +17,7 @@
 * [CHANGE] Renamed `CortexInconsistentConfig` alert to `CortexInconsistentRuntimeConfig` and increased severity to `critical`. #335
 * [CHANGE] Increased `CortexBadRuntimeConfig` alert severity to `critical` and removed support for `cortex_overrides_last_reload_successful` metric (was removed in Cortex 1.3.0). #335
 * [CHANGE] Grafana 'min step' changed to 15s so dashboard show better detail. #340
+* [CHANGE] Replace `CortexRulerFailedEvaluations` with two new alerts: `CortexRulerTooManyFailedPushes` and `CortexRulerTooManyFailedQueries`. #347
 * [CHANGE] Removed `CortexCacheRequestErrors` alert. This alert was not working because the legacy Cortex cache client instrumentation doesn't track errors. #346
 * [CHANGE] Removed `CortexQuerierCapacityFull` alert. #342
 * [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. #317

cortex-mixin/alerts/alerts.libsonnet

@@ -512,20 +512,40 @@
       name: 'ruler_alerts',
       rules: [
         {
-          alert: 'CortexRulerFailedEvaluations',
+          alert: 'CortexRulerTooManyFailedPushes',
           expr: |||
-            sum by (%s, instance, rule_group) (rate(cortex_prometheus_rule_evaluation_failures_total[1m]))
+            100 * (
+            sum by (%s, instance) (rate(cortex_ruler_write_requests_failed_total[1m]))
               /
-            sum by (%s, instance, rule_group) (rate(cortex_prometheus_rule_evaluations_total[1m]))
-              > 0.01
+            sum by (%s, instance) (rate(cortex_ruler_write_requests_total[1m]))
+            ) > 1
+          ||| % [$._config.alert_aggregation_labels, $._config.alert_aggregation_labels],
+          'for': '5m',
+          labels: {
+            severity: 'critical',
+          },
+          annotations: {
+            message: |||
+              Cortex Ruler {{ $labels.instance }} is experiencing {{ printf "%.2f" $value }}% write (push) errors.
+            |||,
+          },
+        },
+        {
+          alert: 'CortexRulerTooManyFailedQueries',
+          expr: |||
+            100 * (
+            sum by (%s, instance) (rate(cortex_ruler_queries_failed_total[1m]))
+              /
+            sum by (%s, instance) (rate(cortex_ruler_queries_total[1m]))
+            ) > 1
           ||| % [$._config.alert_aggregation_labels, $._config.alert_aggregation_labels],
           'for': '5m',
           labels: {
             severity: 'warning',
           },
           annotations: {
             message: |||
-              Cortex Ruler {{ $labels.instance }} is experiencing {{ printf "%.2f" $value }}% errors for the rule group {{ $labels.rule_group }}.
+              Cortex Ruler {{ $labels.instance }} is experiencing {{ printf "%.2f" $value }}% errors while evaluating rules.
             |||,
           },
         },

cortex-mixin/docs/playbooks.md

@@ -146,9 +146,26 @@ More information:
 
 This alert occurs when a ruler is unable to validate whether or not it should claim ownership over the evaluation of a rule group. The most likely cause is that one of the rule ring entries is unhealthy. If this is the case proceed to the ring admin http page and forget the unhealth ruler. The other possible cause would be an error returned the ring client. If this is the case look into debugging the ring based on the in-use backend implementation.
 
-### CortexRulerFailedEvaluations
+### CortexRulerTooManyFailedPushes
 
-_TODO: this playbook has not been written yet._
+This alert fires when rulers cannot push new samples (result of rule evaluation) to ingesters.
+
+In general, pushing samples can fail due to problems with Cortex operations (eg. too many ingesters have crashed, and ruler cannot write samples to them), or due to problems with resulting data (eg. user hitting limit for number of series, out of order samples, etc.).
+This alert fires only for first kind of problems, and not for problems caused by limits or invalid rules.
+
+How to **fix**:
+- Investigate the ruler logs to find out the reason why ruler cannot write samples. Note that ruler logs all push errors, including "user errors", but those are not causing the alert to fire. Focus on problems with ingesters.
+
+### CortexRulerTooManyFailedQueries
+
+This alert fires when rulers fail to evaluate rule queries.
+
+Each rule evaluation may fail due to many reasons, eg. due to invalid PromQL expression, or query hits limits on number of chunks. These are "user errors", and this alert ignores them.
+
+There is a category of errors that is more important: errors due to failure to read data from store-gateways or ingesters. These errors would result in 500 when run from querier. This alert fires if there is too many of such failures.
+
+How to **fix**:
+- Investigate the ruler logs to find out the reason why ruler cannot evaluate queries. Note that ruler logs rule evaluation errors even for "user errors", but those are not causing the alert to fire. Focus on problems with ingesters or store-gateways.
 
 ### CortexRulerMissedEvaluations