You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/kafka/kafka-clickhouse-connect-sink.md
+79-6Lines changed: 79 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -309,20 +309,93 @@ For additional details check out the official [tutorial](https://docs.confluent.
309
309
310
310
ClickHouse Kafka Connect reports runtime metrics via [Java Management Extensions (JMX)](https://www.oracle.com/technical-resources/articles/javase/jmx.html). JMX is enabled in Kafka Connector by default.
The connector integrates with the Kafka Connect framework and exposes metrics for task lifecycle and error tracking.
361
+
362
+
**Task Status Metrics:**
363
+
-`task-count`: Total number of tasks in the connector
364
+
-`running-task-count`: Number of tasks currently running
365
+
-`paused-task-count`: Number of tasks currently paused
366
+
-`failed-task-count`: Number of tasks that have failed
367
+
-`destroyed-task-count`: Number of destroyed tasks
368
+
-`unassigned-task-count`: Number of unassigned tasks
369
+
370
+
Task status values include: `running`, `paused`, `failed`, `destroyed`, `unassigned`
371
+
372
+
**Error Metrics:**
373
+
-`deadletterqueue-produce-failures`: Number of failed DLQ writes
374
+
-`deadletterqueue-produce-requests`: Total DLQ write attempts
375
+
-`last-error-timestamp`: Timestamp of the last error
376
+
-`records-skip-total`: Total number of records skipped due to errors
377
+
-`records-retry-total`: Total number of records that were retried
378
+
-`errors-total`: Total number of errors encountered
379
+
380
+
**Performance Metrics:**
381
+
-`offset-commit-failures`: Number of failed offset commits
382
+
-`offset-commit-avg-time-ms`: Average time for offset commits
383
+
-`offset-commit-max-time-ms`: Maximum time for offset commits
384
+
-`put-batch-avg-time-ms`: Average time to process a batch
385
+
-`put-batch-max-time-ms`: Maximum time to process a batch
386
+
-`source-record-poll-total`: Total records polled
387
+
388
+
#### Monitoring Best Practices {#monitoring-best-practices}
389
+
390
+
1.**Monitor Consumer Lag**: Track `records-lag` per partition to identify processing bottlenecks
391
+
2.**Track Error Rates**: Watch `errors-total` and `records-skip-total` to detect data quality issues
392
+
3.**Observe Task Health**: Monitor task status metrics to ensure tasks are running properly
393
+
4.**Measure Throughput**: Use `records-send-rate` and `byte-rate` to track ingestion performance
394
+
5.**Monitor Connection Health**: Check node-level connection metrics for network issues
395
+
6.**Track Compression Efficiency**: Use `compression-rate` to optimize data transfer
396
+
397
+
For detailed JMX metric definitions and Prometheus integration, see the [jmx-export-connector.yml](https://github.com/ClickHouse/clickhouse-kafka-connect/blob/main/jmx-export-connector.yml) configuration file.
0 commit comments