Skip to content
This repository was archived by the owner on Apr 28, 2025. It is now read-only.

Commit e9dfa71

Browse files
committed
Addressed review comments
Signed-off-by: Marco Pracucci <marco@pracucci.com>
1 parent bb5a40a commit e9dfa71

File tree

1 file changed

+8
-5
lines changed

1 file changed

+8
-5
lines changed

cortex-mixin/docs/playbooks.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -414,13 +414,13 @@ The procedure to investigate it is the same as the one for [`CortexSchedulerQuer
414414

415415
### CortexSchedulerQueriesStuck
416416

417-
This alert fires if Cortex is queries are piling up in the query-scheduler.
417+
This alert fires if queries are piling up in the query-scheduler.
418418

419419
How it **works**:
420420
- A query-frontend API endpoint is called to execute a query
421421
- The query-frontend enqueues the request to the query-scheduler
422-
- The query-scheduler is responsible to dispatch enqueued queries to idle querier workers
423-
- The querier runs the query, sends the response back directly to the query-frontend and notifies the query-scheduler
422+
- The query-scheduler is responsible for dispatching enqueued queries to idle querier workers
423+
- The querier runs the query, sends the response back directly to the query-frontend and notifies the query-scheduler that it can process another query
424424

425425
How to **investigate**:
426426
- Are queriers in a crash loop (eg. OOMKilled)?
@@ -431,8 +431,11 @@ How to **investigate**:
431431
- Is query latency increased?
432432
- An increased latency reduces the number of queries we can run / sec: once all workers are busy, new queries will pile up in the queue
433433
- Temporarily scale up queriers to try to stop the bleed
434-
- Check the `Cortex / Slow Queries` dashboard to see if a specific tenant is running heavy queries
435-
- If it's a multi-tenant Cortex cluster and shuffle-sharing is disabled for queriers, you may consider to enable it only for that specific tenant to reduce its blast radius. To enable queriers shuffle-sharding for a single tenant you need to set the `max_queriers_per_tenant` limit override for the specific tenant (the value should be set to the number of queriers assigned to the tenant).
434+
- Check if a specific tenant is running heavy queries
435+
- Run `sum by (user) (cortex_query_scheduler_queue_length{namespace="<namespace>"}) > 0` to find tenants with enqueued queries
436+
- Check the `Cortex / Slow Queries` dashboard to find slow queries
437+
- On multi-tenant Cortex cluster with **shuffle-sharing for queriers disabled**, you may consider to enable it for that specific tenant to reduce its blast radius. To enable queriers shuffle-sharding for a single tenant you need to set the `max_queriers_per_tenant` limit override for the specific tenant (the value should be set to the number of queriers assigned to the tenant).
438+
- On multi-tenant Cortex cluster with **shuffle-sharding for queriers enabled**, you may consider to temporarily increase the shard size for affected tenants: be aware that this could affect other tenants too, reducing resources available to run other tenant queries. Alternatively, you may choose to do nothing and let Cortex return errors for that given user once the per-tenant queue is full.
436439

437440
### CortexCacheRequestErrors
438441

0 commit comments

Comments
 (0)