Skip to content
This repository was archived by the owner on Apr 28, 2025. It is now read-only.

Commit 2c174a2

Browse files
committed
Add a simple playbook for ingester series limit alert.
Signed-off-by: Callum Styan <callumstyan@gmail.com>
1 parent b320e0f commit 2c174a2

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

cortex-mixin/docs/playbooks.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,9 @@ If nothing obvious from the above, check for increased load:
2626

2727
### CortexIngesterReachingSeriesLimit
2828

29-
_TODO: this playbook has not been written yet._
29+
First check the writes resources dashboard and scaling dashboard. The usual target is 1.5M active series per ingester and a max of 2.5M.
30+
31+
Scaling up the ingesters will help, but it won't resolve the alert immediately, as series are active until the next TSDB Head compaction (every 2h or so). You may also want to temporarily increase the per ingester series limit (this is a runtime reloadable config option) until that next Head compaction occurs.
3032

3133
### CortexIngesterReachingTenantsLimit
3234

0 commit comments

Comments
 (0)