Skip to content

Commit 80bea48

Browse files
authored
Merge pull request #4488 from ClickHouse/tp/clickpipes-s3-advanced-settings
Add new object storage settings for clickpipes
2 parents d46079b + 8ff3410 commit 80bea48

File tree

1 file changed

+20
-2
lines changed
  • docs/integrations/data-ingestion/clickpipes

1 file changed

+20
-2
lines changed

docs/integrations/data-ingestion/clickpipes/index.md

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,11 +86,29 @@ Steps:
8686
## Adjusting ClickPipes advanced settings {#clickpipes-advanced-settings}
8787
ClickPipes provides sensible defaults that cover the requirements of most use cases. If your use case requires additional fine-tuning, you can adjust the following settings:
8888

89-
- **Streaming max insert wait time**: Configures the maximum wait period before inserting data into the ClickHouse cluster. Applies to streaming ClickPipes (e.g., Kafka, Kinesis).
90-
- **Object storage polling interval**: Configures how frequently ClickPipes checks object storage for new data. Applies to object storage ClickPipes (e.g., S3, GCS).
89+
### Object Storage ClickPipes {#clickpipes-advanced-settings-object-storage}
90+
91+
| Setting | Default value | Description |
92+
|------------------------------------|---------------|---------------------------------------------------------------------------------------|
93+
| `Max insert bytes` | 10GB | Number of bytes to process in a single insert batch. |
94+
| `Max file count` | 100 | Maximum number of files to process in a single insert batch. |
95+
| `Max threads` | auto(3) | [Maximum number of concurrent threads](/operations/settings/settings#max_threads) for file processing. |
96+
| `Max insert threads` | 1 | [Maximum number of concurrent insert threads](/operations/settings/settings#max_insert_threads) for file processing. |
97+
| `Min insert block size bytes` | 1GB | [Minimum size of bytes in the block](/operations/settings/settings#min_insert_block_size_bytes) which can be inserted into a table. |
98+
| `Max download threads` | 4 | [Maximum number of concurrent download threads](/operations/settings/settings#max_download_threads). |
99+
| `Object storage polling interval` | 30s | Configures the maximum wait period before inserting data into the ClickHouse cluster. |
100+
| `Parallel distributed insert select` | 2 | [Parallel distributed insert select setting](/operations/settings/settings#parallel_distributed_insert_select). |
101+
| `Parallel view processing` | false | Whether to enable pushing to attached views [concurrently instead of sequentially](/operations/settings/settings#parallel_view_processing). |
102+
| `Use cluster function` | true | Whether to process files in parallel across multiple nodes. |
91103

92104
<Image img={cp_advanced_settings} alt="Advanced settings for ClickPipes" size="lg" border/>
93105

106+
### Streaming ClickPipes {#clickpipes-advanced-settings-streaming}
107+
108+
| Setting | Default value | Description |
109+
|------------------------------------|---------------|---------------------------------------------------------------------------------------|
110+
| `Streaming max insert wait time` | 5s | Configures the maximum wait period before inserting data into the ClickHouse cluster. |
111+
94112
## Error reporting {#error-reporting}
95113
ClickPipes will store errors in two separate tables depending on the type of error encountered during the ingestion process.
96114
### Record Errors {#record-errors}

0 commit comments

Comments
 (0)