You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/clickpipes/index.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -39,7 +39,7 @@ import Image from '@theme/IdealImage';
39
39
| Confluent Cloud | <Confluentsvg class="image" alt="Confluent Cloud logo" style={{width: '3rem'}}/> |Streaming| Stable | Unlock the combined power of Confluent and ClickHouse Cloud through our direct integration. |
40
40
| Redpanda | <Imageimg={redpanda_logo}size="logo"alt="Redpanda logo"/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Redpanda into ClickHouse Cloud. |
41
41
| AWS MSK | <Msksvg class="image" alt="AWS MSK logo" style={{width: '3rem', 'height': '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from AWS MSK into ClickHouse Cloud. |
42
-
| Azure Event Hubs | <Azureeventhubssvg class="image" alt="Azure Event Hubs logo" style={{width: '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Azure Event Hubs into ClickHouse Cloud. |
42
+
| Azure Event Hubs | <Azureeventhubssvg class="image" alt="Azure Event Hubs logo" style={{width: '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from Azure Event Hubs into ClickHouse Cloud. See the [Azure Event Hubs FAQ](/integrations/clickpipes/kafka/faq/#azure-eventhubs) for guidance. |
43
43
| WarpStream | <Warpstreamsvg class="image" alt="WarpStream logo" style={{width: '3rem'}}/> |Streaming| Stable | Configure ClickPipes and start ingesting streaming data from WarpStream into ClickHouse Cloud. |
44
44
| Amazon S3 | <S3svg class="image" alt="Amazon S3 logo" style={{width: '3rem', height: 'auto'}}/> |Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
45
45
| Google Cloud Storage | <Gcssvg class="image" alt="Google Cloud Storage logo" style={{width: '3rem', height: 'auto'}}/> |Object Storage| Stable | Configure ClickPipes to ingest large volumes of data from object storage. |
@@ -101,7 +101,7 @@ If ClickPipes cannot connect to a data source after 15 min or to a destination a
101
101
102
102
-**Does using ClickPipes incur an additional cost?**
103
103
104
-
ClickPipes is billed on two dimensions: Ingested Data and Compute. The full details of the pricing are available on [this page](/cloud/manage/jan-2025-faq/pricing-dimensions#clickpipes-pricing-faq). Running ClickPipes might also generate an indirect compute and storage cost on the destination ClickHouse Cloud service similar to any ingest workload.
104
+
ClickPipes is billed on two dimensions: Ingested Data and Compute. The full details of the pricing are available on [this page](/cloud/manage/billing/overview#clickpipes-for-streaming-object-storage). Running ClickPipes might also generate an indirect compute and storage cost on the destination ClickHouse Cloud service similar to any ingest workload.
105
105
106
106
-**Is there a way to handle errors or failures when using ClickPipes for Kafka?**
@@ -12,13 +12,12 @@ ClickPipes supports schema registries for Avro data streams.
12
12
13
13
## Supported registries for Kafka ClickPipes {#supported-schema-registries}
14
14
15
-
Schema registries that use the Confluent Schema Registry API are supported. This includes:
16
-
- Confluent Kafka and Cloud
17
-
- Redpanda
18
-
- AWS MSK
19
-
- Upstash
15
+
Schema registries that are API-compatible with the Confluent Schema Registry are supported. This includes:
20
16
21
-
ClickPipes is not currently compatible with the AWS Glue Schema registry or the Azure Schema Registry.
17
+
- Confluent Schema Registry
18
+
- Redpanda Schema Registry
19
+
20
+
ClickPipes does not support AWS Glue Schema Registry or Azure Schema Registry yet. If you require support for these schema registries, [reach out to our team](https://clickhouse.com/company/contact?loc=clickpipes).
22
21
23
22
## Configuration {#schema-registry-configuration}
24
23
@@ -31,7 +30,7 @@ ClickPipes with Avro data require a schema registry. This can be configured in o
31
30
32
31
## How it works {#how-schema-registries-work}
33
32
34
-
ClickPipes dynamically retrieves and applies the Avro schema from the configured Schema Registry.
33
+
ClickPipes dynamically retrieves and applies the Avro schema from the configured schema registry.
35
34
- If there's a schema id embedded in the message, it will use that to retrieve the schema.
36
35
- If there's no schema id embedded in the message, it will use the schema id or subject name specified in the ClickPipe configuration to retrieve the schema.
37
36
- If the message is written without an embedded schema id, and no schema id or subject name is specified in the ClickPipe configuration, then the schema will not be retrieved and the message will be skipped with a `SOURCE_SCHEMA_ERROR` logged in the ClickPipes errors table.
Copy file name to clipboardExpand all lines: docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md
+13-36Lines changed: 13 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
2
sidebar_label: 'FAQ'
3
-
description: 'Frequently asked questions about Kafka ClickPipes'
3
+
description: 'Frequently asked questions about ClickPipes for Kafka'
4
4
slug: /integrations/clickpipes/kafka/faq
5
5
sidebar_position: 1
6
6
title: 'Kafka ClickPipes FAQ'
@@ -23,7 +23,7 @@ ClickPipes uses a dedicated architecture running the Kafka Consumer API to read
23
23
24
24
The Kafka Table engine is a ClickHouse core capability that implements a "pull model" where the ClickHouse server itself connects to Kafka, pulls events then writes them locally.
25
25
26
-
ClickPipes is a separate cloud service that runs independently of the ClickHouse Service, it connects to Kafka (or other data sources) and pushes events to an associated ClickHouse Cloud service. This decoupled architecture allows for superior operational flexibility, clear separation of concerns, scalable ingestion, graceful failure management, extensibility and more.
26
+
ClickPipes is a separate cloud service that runs independently of the ClickHouse service. It connects to Kafka (or other data sources) and pushes events to an associated ClickHouse Cloud service. This decoupled architecture allows for superior operational flexibility, clear separation of concerns, scalable ingestion, graceful failure management, extensibility, and more.
27
27
</details>
28
28
29
29
<details>
@@ -63,77 +63,54 @@ Horizontal scaling adds more replicas to increase throughput, while vertical sca
63
63
This can be configured during ClickPipe creation, or at any other point under **Settings** -> **Advanced Settings** -> **Scaling**.
64
64
</details>
65
65
66
-
### Upstash {#upstash}
67
-
68
-
<details>
69
-
70
-
<summary>Does ClickPipes support Upstash?</summary>
71
-
72
-
Yes. The Upstash Kafka product entered into a deprecation period on 11th September 2024 for 6 months. Existing customers can continue to use ClickPipes with their existing Upstash Kafka brokers using the generic Kafka tile on the ClickPipes user interface. Existing Upstash Kafka ClickPipes are unaffected before the deprecation notice. When the the deprecation period is up the ClickPipe will stop functioning.
73
-
</details>
74
-
75
-
<details>
76
-
77
-
<summary>Does ClickPipes support Upstash schema registry?</summary>
78
-
79
-
No. ClickPipes is not Upstash Kafka schema registry compatible.
80
-
</details>
81
-
82
-
<details>
83
-
84
-
<summary>Does ClickPipes support the Upstash QStash Workflow?</summary>
85
-
86
-
No. Unless a Kafka compatible surface is introduced in QStash Workflow it will not work with Kafka ClickPipes.
87
-
</details>
88
-
89
-
### Azure EventHubs {#azure-eventhubs}
66
+
### Azure Event Hubs {#azure-eventhubs}
90
67
91
68
<details>
92
69
93
70
<summary>Does the Azure Event Hubs ClickPipe work without the Kafka surface?</summary>
94
71
95
-
No. ClickPipes requires the Azure Event Hubs to have the Kafka surface enabled. The Kafka protocol is supported for their Standard, Premium and Dedicated SKU only pricing tiers.
72
+
No. ClickPipes requires the Event Hubs namespace to have the Kafka surface enabled. This is only available in tiers above **basic**. See the [Azure Event Hubs documentation](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-quickstart-kafka-enabled-event-hubs?tabs=passwordless#create-an-azure-event-hubs-namespace) for more information.
96
73
</details>
97
74
98
75
<details>
99
76
100
-
<summary>Does Azure schema registry work with ClickPipes</summary>
77
+
<summary>Does Azure Schema Registry work with ClickPipes?</summary>
101
78
102
-
No. ClickPipes is not currently Event Hubs Schema Registry compatible.
79
+
No. ClickPipes only supports schema registries that are API-compatible with the Confluent Schema Registry, which is not the case for Azure Schema Registry. If you require support for this schema registry, [reach out to our team](https://clickhouse.com/company/contact?loc=clickpipes).
103
80
</details>
104
81
105
82
<details>
106
83
107
84
<summary>What permissions does my policy need to consume from Azure Event Hubs?</summary>
108
85
109
-
To list topics and consume event, the shared access policy that is given to ClickPipes will at minimum require a 'Listen' claim.
86
+
To list topics and consume events, the shared access policy that is given to ClickPipes requires, at minimum, a 'Listen' claim.
110
87
</details>
111
88
112
89
<details>
113
90
114
91
<summary>Why is my Event Hubs not returning any data?</summary>
115
92
116
-
If your ClickHouse instance is in a different region or continent from your Event Hubs deployment, you may experience timeouts when onboarding your ClickPipes, and higher-latency when consuming data from the Event Hub. It is considered a best practice to locate your ClickHouse Cloud deployment and Azure Event Hubs deployment in cloud regions located close to each other to avoid adverse performance.
93
+
If your ClickHouse instance is in a different region or continent from your Event Hubs deployment, you may experience timeouts when onboarding your ClickPipes, and higher-latency when consuming data from the Event Hub. We recommend deploying ClickHouse Cloud and Azure Event Hubs in the same cloud region, or regions located close to each other, to avoid performance overhead.
117
94
</details>
118
95
119
96
<details>
120
97
121
98
<summary>Should I include the port number for Azure Event Hubs?</summary>
122
99
123
-
Yes. ClickPipes expects you to include your port number for the Kafka surface, which should be `:9093`.
100
+
Yes. ClickPipes expects you to include the port number for the Kafka surface, which should be `:9093`.
124
101
</details>
125
102
126
103
<details>
127
104
128
-
<summary>Are the ClickPipes IPs still relevant for Azure Event Hubs?</summary>
105
+
<summary>Are ClickPipes IPs still relevant for Azure Event Hubs?</summary>
129
106
130
-
Yes. If you restrict traffic to your Event Hubs instance please add the [documented static NAT IPs](../
131
-
/index.md#list-of-static-ips).
107
+
Yes. To restrict traffic to your Event Hubs instance, please add the [documented static NAT IPs](../
108
+
/index.md#list-of-static-ips) to .
132
109
133
110
</details>
134
111
135
112
<details>
136
113
<summary>Is the connection string for the Event Hub, or is it for the Event Hub namespace?</summary>
137
114
138
-
Both will work, however, we recommend using a shared access policy at the namespace level to retrieve samples from multiple Event Hubs.
115
+
Both work. We strongly recommend using a shared access policy at the **namespace level** to retrieve samples from multiple Event Hubs.
0 commit comments