Skip to content

Commit c612ec5

Browse files
authored
DB ClickPipes: document Completed state (#4361)
1 parent 8cc85bf commit c612ec5

File tree

3 files changed

+18
-6
lines changed

3 files changed

+18
-6
lines changed

docs/integrations/data-ingestion/clickpipes/mongodb/lifecycle.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,11 @@ After a pipe is provisioned, it enters the `Setup` state. This state is where we
1919

2020
## Snapshot {#snapshot}
2121

22-
Once setup is complete, we enter the `Snapshot` state. `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source MongoDB collections and load them into ClickHouse. Retention setting for the oplog should account for initial load time. The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
22+
Once setup is complete, we enter the `Snapshot` state (unless it's a CDC-only pipe, which would transition to `Running`). `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source MongoDB collections and load them into ClickHouse. Retention setting for the oplog should account for initial load time. The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
2323

2424
## Running {#running}
2525

26-
Once the initial load is complete, the pipe enters the `Running` state. This is where the pipe begins `Change-Data Capture`. In this state, we start streaming changes from the source MongoDB cluster to ClickHouse. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
26+
Once the initial load is complete, the pipe enters the `Running` state (unless it's a snapshot-only pipe, which would transition to `Completed`). This is where the pipe begins `Change-Data Capture`. In this state, we start streaming changes from the source MongoDB cluster to ClickHouse. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
2727

2828
## Paused {#paused}
2929

@@ -48,6 +48,10 @@ This state is coming soon. If you're using our [OpenAPI](https://clickhouse.com/
4848
:::
4949
This state indicates the pipe is in the phase of resync where it is performing an atomic swap of the _resync tables with the original tables. More information on resync can be found in the [resync documentation](./resync).
5050

51+
## Completed {#completed}
52+
53+
This state applies to snapshot-only pipes and indicates that the snapshot has been completed and there's no more work to do.
54+
5155
## Failed {#failed}
5256

5357
If there is an irrecoverable error in the pipe, it will enter the `Failed` state. You can reach out to support or [resync](./resync) your pipe to recover from this state.

docs/integrations/data-ingestion/clickpipes/mysql/lifecycle.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,11 @@ After a pipe is provisioned, it enters the `Setup` state. This state is where we
1919

2020
## Snapshot {#snapshot}
2121

22-
Once setup is complete, we enter the `Snapshot` state. `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source MySQL tables and load them into ClickHouse. Retention setting for binary logs should account for initial load time. For more information on initial load, see the [parallel initial load documentation](./parallel_initial_load). The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
22+
Once setup is complete, we enter the `Snapshot` state (unless it's a CDC-only pipe, which would transition to `Running`). `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source MySQL tables and load them into ClickHouse. Retention setting for binary logs should account for initial load time. For more information on initial load, see the [parallel initial load documentation](./parallel_initial_load). The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
2323

2424
## Running {#running}
2525

26-
Once the initial load is complete, the pipe enters the `Running` state. This is where the pipe begins `Change-Data Capture`. In this state, we start reading binary logs from the source database and sync the data to ClickHouse in batches. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
26+
Once the initial load is complete, the pipe enters the `Running` state (unless it's a snapshot-only pipe, which would transition to `Completed`). This is where the pipe begins `Change-Data Capture`. In this state, we start reading binary logs from the source database and sync the data to ClickHouse in batches. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
2727

2828
## Paused {#paused}
2929

@@ -48,6 +48,10 @@ This state is coming soon. If you're using our [OpenAPI](https://clickhouse.com/
4848
:::
4949
This state indicates the pipe is in the phase of resync where it is performing an atomic swap of the _resync tables with the original tables. More information on resync can be found in the [resync documentation](./resync).
5050

51+
## Completed {#completed}
52+
53+
This state applies to snapshot-only pipes and indicates that the snapshot has been completed and there's no more work to do.
54+
5155
## Failed {#failed}
5256

5357
If there is an irrecoverable error in the pipe, it will enter the `Failed` state. You can reach out to support or [resync](./resync) your pipe to recover from this state.

docs/integrations/data-ingestion/clickpipes/postgres/lifecycle.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,11 @@ After a pipe is provisioned, it enters the `Setup` state. This state is where we
1919

2020
## Snapshot {#snapshot}
2121

22-
Once setup is complete, we enter the `Snapshot` state. `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source database tables and load them into ClickHouse. This does not use logical replication, but the replication slot is created at this step, therefore your `max_slot_wal_keep_size` and storage parameters should account for slot growth during initial load. For more information on initial load, see the [parallel initial load documentation](./parallel_initial_load). The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
22+
Once setup is complete, we enter the `Snapshot` state (unless it's a CDC-only pipe, which would transition to `Running`). `Snapshot`, `Initial Snapshot` and `Initial Load` (more common) are interchangeable terms. In this state, we take a snapshot of the source database tables and load them into ClickHouse. This does not use logical replication, but the replication slot is created at this step, therefore your `max_slot_wal_keep_size` and storage parameters should account for slot growth during initial load. For more information on initial load, see the [parallel initial load documentation](./parallel_initial_load). The pipe will also enter the `Snapshot` state when a resync is triggered or when new tables are added to an existing pipe.
2323

2424
## Running {#running}
2525

26-
Once the initial load is complete, the pipe enters the `Running` state. This is where the pipe begins `Change-Data Capture`. In this state, we start logical replication from the source database to ClickHouse. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
26+
Once the initial load is complete, the pipe enters the `Running` state (unless it's a snapshot-only pipe, which would transition to `Completed`). This is where the pipe begins `Change-Data Capture`. In this state, we start logical replication from the source database to ClickHouse. For information on controlling CDC, see [the doc on controlling CDC](./sync_control).
2727

2828
## Paused {#paused}
2929

@@ -48,6 +48,10 @@ This state is coming soon. If you're using our [OpenAPI](https://clickhouse.com/
4848
:::
4949
This state indicates the pipe is in the phase of resync where it is performing an atomic swap of the _resync tables with the original tables. More information on resync can be found in the [resync documentation](./resync).
5050

51+
## Completed {#completed}
52+
53+
This state applies to snapshot-only pipes and indicates that the snapshot has been completed and there's no more work to do.
54+
5155
## Failed {#failed}
5256

5357
If there is an irrecoverable error in the pipe, it will enter the `Failed` state. You can reach out to support or [resync](./resync) your pipe to recover from this state.

0 commit comments

Comments
 (0)