Skip to content

Commit 26e433f

Browse files
dhtclkdhtclk
committed
monitoring guides with snippets for cloud and self-managed
1 parent d60d4e1 commit 26e433f

File tree

6 files changed

+218
-0
lines changed

6 files changed

+218
-0
lines changed

‎docs/_snippets/_community_monitoring.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
## Community Monitoring Solutions {#community-monitoring}
2+
3+
The ClickHouse community has developed comprehensive monitoring solutions that integrate with popular observability stacks. [ClickHouse Monitoring](https://github.com/duyet/clickhouse-monitoring) provides a complete monitoring setup with pre-built dashboards. This open source project offers a quick-start approach for teams looking to implement ClickHouse monitoring with established best practices and proven dashboard configurations.
4+
5+
:::note
6+
Like other direct database monitoring approaches, this solution queries ClickHouse system tables directly, which prevents instances from idling and impacts cost optimization.
7+
:::

‎docs/_snippets/_direct_observability_integration_options.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
import Image from '@theme/IdealImage';
2+
import AdvancedDashboard from '@site/static/images/cloud/manage/monitoring/advanced_dashboard.png';
3+
import NativeAdvancedDashboard from '@site/static/images/cloud/manage/monitoring/native_advanced_dashboard.png';
4+
5+
### Direct Grafana plugin integration {#direct-grafana}
6+
7+
The ClickHouse data source plugin for Grafana enables visualization and exploration of data directly from ClickHouse using system tables. This approach works well for monitoring performance and creating custom dashboards for detailed system analysis.
8+
For plugin installation and configuration details, see the ClickHouse data source plugin. For a complete monitoring setup using the Prometheus-Grafana mix-in with pre-built dashboards and alerting rules, see Monitor ClickHouse with the new Prometheus-Grafana mix-in.
9+
10+
### Direct Datadog Integration {#direct-datadog}
11+
12+
Datadog offers a Clickhouse Monitoring plugin for its agent which queries system tables directly. This integration provides comprehensive database monitoring with cluster awareness through clusterAllReplicas functionality.
13+
Note: This integration is not recommended for ClickHouse Cloud deployments due to incompatibility with cost-optimizing idle behavior and operational limitations of the cloud proxy layer.
14+
15+
### Using system tables directly {#system-tables}
16+
17+
Users can perform deep query performance analysis by connecting to ClickHouse system tables, particularly system.query_log and querying directly. Using either the SQL console or clickhouse client, teams can identify slow queries, analyze resource usage, and track usage patterns across the organization.
18+
19+
**Query Performance Analysis**
20+
21+
Users can use the system table query logs to perform Query Performance Analysis.
22+
23+
**Example query**: Find the top 5 long-running queries across all cluster replicas:
24+
25+
```sql
26+
SELECT
27+
type,
28+
event_time,
29+
query_duration_ms,
30+
query,
31+
read_rows,
32+
tables
33+
FROM clusterAllReplicas(default, system.query_log)
34+
WHERE event_time >= (now() - toIntervalMinute(60)) AND type='QueryFinish'
35+
ORDER BY query_duration_ms DESC
36+
LIMIT 5
37+
FORMAT VERTICAL
38+
```

‎docs/_snippets/_observability_integration_options.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
import Image from '@theme/IdealImage';
2+
import AdvancedDashboard from '@site/static/images/cloud/manage/monitoring/advanced_dashboard.png';
3+
import NativeAdvancedDashboard from '@site/static/images/cloud/manage/monitoring/native_advanced_dashboard.png';
4+
5+
## Integration examples {#examples}
6+
7+
External integration allows organizations to maintain established monitoring workflows, leverage existing team expertise with familiar tools, and integrate ClickHouse monitoring with broader infrastructure observability without disrupting current processes or requiring significant retraining investments.
8+
Teams can apply existing alerting rules and escalation procedures to ClickHouse metrics, while correlating database performance with application and infrastructure health within a unified observability platform. This approach maximizes ROI on current monitoring investments and enables faster troubleshooting through consolidated dashboards and familiar tooling interfaces.
9+
10+
### Grafana Cloud Monitoring {#grafana}
11+
12+
Grafana provides ClickHouse monitoring through both direct plugin integration and Prometheus-based approaches. The Prometheus endpoint integration maintains operational separation between monitoring and production workloads while enabling visualization within existing Grafana Cloud infrastructure. See Grafana's ClickHouse documentation for configuration guidance.
13+
Datadog monitoring
14+
Datadog is developing a dedicated API integration that will provide proper cloud service monitoring while respecting service idling behavior. In the interim, teams can use the OpenMetrics integration approach via ClickHouse Prometheus endpoints for operational separation and cost-efficient monitoring. For configuration guidance, see Datadog's Prometheus and OpenMetrics integration documentation.
15+
16+
### ClickStack {#clickstack}
17+
18+
ClickStack is ClickHouse's recommended observability solution for deep system analysis and debugging, providing a unified platform for logs, metrics, and traces using ClickHouse as the storage engine. This approach relies on HyperDX, the ClickStack UI, connecting directly to the system tables inside your ClickHouse instance.
19+
HyperDX ships with a ClickHouse focused dashboard with tabs for Selects, Inserts, and Infrastructure. Teams can also use Lucene or SQL syntax to search system tables and logs, as well as create custom visualizations via Chart Explorer for detailed system analysis.
20+
This approach is ideal for debugging complex issues, performance analysis, and deep system introspection rather than real-time production alerting.
21+
22+
:::note
23+
Note that this approach will wake idle services as HyperDX queries the system tables directly.
24+
:::

‎docs/use-cases/observability/cloud-monitoring.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
---
2+
slug: /use-cases/observability/cloud-monitoring
3+
title: 'ClickHouse Cloud Monitoring'
4+
description: 'ClickHouse Cloud Monitoring Guide'
5+
doc_type: 'guide'
6+
---
7+
8+
import AdvancedDashboard from '@site/static/images/cloud/manage/monitoring/advanced_dashboard.png';
9+
import NativeAdvancedDashboard from '@site/static/images/cloud/manage/monitoring/native_advanced_dashboard.png';
10+
import Image from '@theme/IdealImage';
11+
import ObservabilityIntegrations from '@site/docs/_snippets/_observability_integration_options.md';
12+
import DirectIntegrations from '@site/docs/_snippets/_direct_observability_integration_options.md';
13+
import CommunityMonitoring from '@site/docs/_snippets/_community_monitoring.md';
14+
15+
# ClickHouse Cloud Monitoring {#cloud-monitoring}
16+
17+
This guide provides enterprise teams evaluating ClickHouse Cloud with comprehensive information on monitoring and observability capabilities for production deployments. Enterprise customers frequently ask about out-of-the-box monitoring features, integration with existing observability stacks including tools like Datadog and AWS CloudWatch, and how ClickHouse’s monitoring compares to self-hosted deployments.
18+
19+
## Advanced Observability Dashboard {#advanced-observability}
20+
21+
ClickHouse Cloud provides comprehensive monitoring through built-in dashboard interfaces accessible via the Monitoring section. These dashboards visualize system and performance metrics in real-time without requiring additional setup and serve as the primary tools for real-time production monitoring within ClickHouse Cloud.
22+
23+
- **Advanced Dashboard**: The main dashboard interface accessible via Monitoring → Advanced dashboard provides real-time visibility into query rates, resource usage, system health, and storage performance. This dashboard doesn't require separate authentication, won't prevent instances from idling, and doesn't add query load to your production system. Each visualization is powered by customizable SQL queries, with out-of-the-box charts grouped into ClickHouse-specific, system health, and Cloud-specific metrics. Users can extend monitoring by creating custom queries directly in the SQL console.
24+
25+
:::note
26+
27+
Accessing these metrics does not issue a query to the underlying service and will not wake idle services.
28+
29+
:::
30+
31+
<Image img={AdvancedDashboard} size="lg" alt="Advanced dashboard"/>
32+
33+
Users looking to extend these visualizations can use the Dashboards feature in ClickHouse Cloud, querying system tables directly.
34+
35+
- **Native Advanced Dashboard**: An alternative dashboard interface accessible through "You can still access the native advanced dashboard" within the Monitoring section. This opens in a separate tab with authentication and provides an alternative UI for system and service health monitoring. This dashboard allows advanced analytics, where users can modify the underlying SQL queries.
36+
37+
<Image img={NativeAdvancedDashboard} size="lg" alt="Advanced dashboard"/>
38+
39+
Both dashboards offer immediate visibility into service health and performance without external dependencies, distinguishing them from external debugging-focused tools like ClickStack.
40+
41+
For detailed dashboard features and available metrics, see the [advanced dashboard documentation](/cloud/manage/monitor/advanced-dashboard).
42+
43+
## Query Insights and Resource Monitoring {#query-insights}
44+
45+
ClickHouse Cloud includes additional monitoring capabilities:
46+
47+
- Query Insights: Built-in interface for query performance analysis and troubleshooting
48+
- Resource Utilization Dashboard: Tracks memory, CPU allocation, and data transfer patterns
49+
50+
See the [query insights](/cloud/get-started/query-insights) and [resource utilization](/operations/monitoring#resource-utilization) documentation for detailed features.
51+
52+
## Prometheus-compatible metrics endpoint {#prometheus}
53+
54+
ClickHouse Cloud provides a Prometheus endpoint. This allows users to maintain current workflows, leverage existing team expertise, and integrate ClickHouse metrics into enterprise monitoring platforms including Grafana, Datadog, and other Prometheus-compatible tools.
55+
56+
The organization-level endpoint federates metrics from all services, while per-service endpoints provide granular monitoring. Key features include:
57+
- Filtered metrics option: The optional filtered_metrics=true parameter reduces payload from 1000+ available metrics to 125 'mission critical' metrics for cost optimization and easier monitoring focus
58+
- Cached metric delivery: Uses materialized views refreshed every minute to minimize query load on production systems
59+
60+
:::note
61+
62+
This approach respects service idling behavior, allowing for cost optimization when services are not actively processing queries. This API endpoint relies on ClickHouse Cloud API credentials. For complete endpoint configuration details, see the cloud [Prometheus documentation](/integrations/prometheus).
63+
64+
:::
65+
66+
<ObservabilityIntegrations/>
67+
68+
### ClickStack {#clickstack}
69+
70+
ClickStack is ClickHouse's recommended observability solution for deep system analysis and debugging, providing a unified platform for logs, metrics, and traces using ClickHouse as the storage engine. This approach relies on HyperDX, the ClickStack UI, connecting directly to the system tables inside your ClickHouse instance.
71+
HyperDX ships with a ClickHouse focused dashboard with tabs for Selects, Inserts, and Infrastructure. Teams can also use Lucene or SQL syntax to search system tables and logs, as well as create custom visualizations via Chart Explorer for detailed system analysis.
72+
This approach is ideal for debugging complex issues, performance analysis, and deep system introspection rather than real-time production alerting.
73+
74+
:::note
75+
Note that this approach will wake idle services as HyperDX queries the system tables directly.
76+
:::
77+
78+
### ClickStack Deployment Options {#clickstack-deployment}
79+
80+
- **HyperDX in Clickhouse Cloud** (private preview): HyperDx can be launched on any Clickhouse Cloud service.
81+
- [Helm](/use-cases/observability/clickstack/deployment/helm): Recommended for Kubernetes-based debugging environments. Supports integration with ClickHouse Cloud and allows for environment-specific configuration, resource limits, and scaling via `values.yaml`.
82+
- [Docker Compose](/use-cases/observability/clickstack/deployment/docker-compose): Deploys each component (ClickHouse, HyperDX, OTel collector, MongoDB) individually. Users can modify the compose file to remove any unused components when integrating with ClickHouse Cloud, specifically ClickHouse and the Open Telemetry Collector.
83+
- [HyperDX Only](/use-cases/observability/clickstack/deployment/hyperdx-only): Standalone HyperDX container.
84+
85+
For complete deployment options and architecture details, see the [ClickStack documentation](/use-cases/observability/clickstack/overview) and [data ingestion guide](/use-cases/observability/clickstack/ingesting-data/overview).
86+
87+
:::note
88+
Users can also collect metrics from the ClickHouse Cloud Prometheus endpoint via an OpenTelemetry Collector and forward them to a separate ClickStack deployment for visualization.
89+
:::
90+
91+
<DirectIntegrations/>
92+
93+
<CommunityMonitoring/>
94+
95+
## System Impact Considerations {#system-impact}
96+
97+
All of the above approaches use a mixture of either relying on Prometheus endpoints, being managed by ClickHouse Cloud, or querying of system tables directly.
98+
The latter of these options relies on querying the production ClickHouse service. This adds query load to the system under observation and prevents ClickHouse Cloud instances from idling, impacting cost optimization. Additionally, if the production system fails, monitoring may also be affected since they are coupled. This approach works well for deep introspection and debugging but is less appropriate for real-time production monitoring. Consider these trade-offs between detailed system analysis capabilities and operational overhead when evaluating direct Grafana integration versus the external tool integration approaches discussed in the following section.

‎docs/use-cases/observability/self-managed-monitoring.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
slug: /use-cases/observability/oss-monitoring
3+
title: 'Self-Managed Monitoring'
4+
description: 'Self-Managed Monitoring Guide'
5+
doc_type: 'guide'
6+
---
7+
8+
import ObservabilityIntegrations from '@site/docs/_snippets/_observability_integration_options.md';
9+
import DirectIntegrations from '@site/docs/_snippets/_direct_observability_integration_options.md';
10+
import CommunityMonitoring from '@site/docs/_snippets/_community_monitoring.md';
11+
12+
# Self-Managed Monitoring {#cloud-monitoring}
13+
14+
This guide provides enterprise teams evaluating ClickHouse open-source with comprehensive information on monitoring and observability capabilities for production deployments. Enterprise customers frequently ask about out-of-the-box monitoring features, integration with existing observability stacks including tools like Datadog and AWS CloudWatch, and how ClickHouse’ss monitoring compares to self-hosted deployments.
15+
16+
### Prometheus-Based Integration Architecture {#prometheus}
17+
ClickHouse exposes Prometheus-compatible metrics through different endpoints depending on your deployment model, each with distinct operational characteristics:
18+
19+
**Self-Managed/OSS ClickHouse**
20+
21+
Direct server Prometheus endpoint accessible via the standard /metrics endpoint on your ClickHouse server. This approach provides:
22+
- Complete metric exposure: Full range of available ClickHouse metrics without built-in filtering
23+
- Real-time metrics: Generated directly from system tables when scraped
24+
25+
**Direct system access**
26+
27+
Queries production system tables, which adds monitoring load and prevents cost-saving idle states
28+
29+
<ObservabilityIntegrations/>
30+
31+
### ClickStack Deployment Options {#clickstack-deployment}
32+
33+
- [Helm](/use-cases/observability/clickstack/deployment/helm): Recommended for Kubernetes-based debugging environments. Allows for environment-specific configuration, resource limits, and scaling via `values.yaml`.
34+
- [Docker Compose](/use-cases/observability/clickstack/deployment/docker-compose): Deploys each component (ClickHouse, HyperDX, OTel collector, MongoDB) individually.
35+
- [HyperDX Only](/use-cases/observability/clickstack/deployment/hyperdx-only): Standalone HyperDX container.
36+
37+
For complete deployment options and architecture details, see the [ClickStack documentation](/use-cases/observability/clickstack/overview) and [data ingestion guide](/use-cases/observability/clickstack/ingesting-data/overview).
38+
39+
<DirectIntegrations/>
40+
41+
<CommunityMonitoring/>

‎sidebars.js‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,16 @@ const sidebars = {
128128
collapsible: true,
129129
link: { type: "doc", id: "use-cases/observability/index" },
130130
items: [
131+
{
132+
type: "category",
133+
label: "Capabilities",
134+
collapsed: true,
135+
collapsible: true,
136+
items: [
137+
"use-cases/observability/cloud-monitoring",
138+
"use-cases/observability/self-managed-monitoring"
139+
]
140+
},
131141
{
132142
type: "category",
133143
label: "ClickStack",

0 commit comments

Comments
 (0)