Skip to content

Commit 991b20d

Browse files
committed
clickpipes imgs
1 parent 65e091c commit 991b20d

File tree

9 files changed

+63
-58
lines changed

9 files changed

+63
-58
lines changed

docs/integrations/data-ingestion/clickpipes/kinesis.md

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ import cp_success from '@site/static/images/integrations/data-ingestion/clickpip
1818
import cp_remove from '@site/static/images/integrations/data-ingestion/clickpipes/cp_remove.png';
1919
import cp_destination from '@site/static/images/integrations/data-ingestion/clickpipes/cp_destination.png';
2020
import cp_overview from '@site/static/images/integrations/data-ingestion/clickpipes/cp_overview.png';
21+
import Image from '@theme/IdealImage';
22+
2123

2224
# Integrating Amazon Kinesis with ClickHouse Cloud
2325
## Prerequisite {#prerequisite}
@@ -27,57 +29,57 @@ You have familiarized yourself with the [ClickPipes intro](./index.md) and setup
2729

2830
1. Access the SQL Console for your ClickHouse Cloud Service.
2931

30-
<img src={cp_service} alt="ClickPipes service" />
32+
<Image img={cp_service} alt="ClickPipes service" size="lg"/>
3133

3234
2. Select the `Data Sources` button on the left-side menu and click on "Set up a ClickPipe"
3335

34-
<img src={cp_step0} alt="Select imports" />
36+
<Image img={cp_step0} alt="Select imports" size="lg" />
3537

3638
3. Select your data source.
3739

38-
<img src={cp_step1} alt="Select data source type" />
40+
<Image img={cp_step1} alt="Select data source type" size="lg" />
3941

4042
4. Fill out the form by providing your ClickPipe with a name, a description (optional), your IAM role or credentials, and other connection details.
4143

42-
<img src={cp_step2_kinesis} alt="Fill out connection details" />
44+
<Image img={cp_step2_kinesis} alt="Fill out connection details" size="lg" />
4345

4446
5. Select Kinesis Stream and starting offset. The UI will display a sample document from the selected source (Kafka topic, etc). You can also enable Enhanced Fan-out for Kinesis streams to improve the performance and stability of your ClickPipe (More information on Enhanced Fan-out can be found [here](https://aws.amazon.com/blogs/aws/kds-enhanced-fanout))
4547

46-
<img src={cp_step3_kinesis} alt="Set data format and topic" />
48+
<Image img={cp_step3_kinesis} alt="Set data format and topic" size="lg" />
4749

4850
6. In the next step, you can select whether you want to ingest data into a new ClickHouse table or reuse an existing one. Follow the instructions in the screen to modify your table name, schema, and settings. You can see a real-time preview of your changes in the sample table at the top.
4951

50-
<img src={cp_step4a} alt="Set table, schema, and settings" />
52+
<Image img={cp_step4a} alt="Set table, schema, and settings" size="lg" />
5153

5254
You can also customize the advanced settings using the controls provided
5355

54-
<img src={cp_step4a3} alt="Set advanced controls" />
56+
<Image img={cp_step4a3} alt="Set advanced controls" size="lg" />
5557

5658
7. Alternatively, you can decide to ingest your data in an existing ClickHouse table. In that case, the UI will allow you to map fields from the source to the ClickHouse fields in the selected destination table.
5759

58-
<img src={cp_step4b} alt="Use an existing table" />
60+
<Image img={cp_step4b} alt="Use an existing table" size="lg" />
5961

6062
8. Finally, you can configure permissions for the internal ClickPipes user.
6163

6264
**Permissions:** ClickPipes will create a dedicated user for writing data into a destination table. You can select a role for this internal user using a custom role or one of the predefined role:
6365
- `Full access`: with the full access to the cluster. It might be useful if you use Materialized View or Dictionary with the destination table.
6466
- `Only destination table`: with the `INSERT` permissions to the destination table only.
6567

66-
<img src={cp_step5} alt="Permissions" />
68+
<Image img={cp_step5} alt="Permissions" />
6769

6870
9. By clicking on "Complete Setup", the system will register you ClickPipe, and you'll be able to see it listed in the summary table.
6971

70-
<img src={cp_success} alt="Success notice" />
72+
<Image img={cp_success} alt="Success notice" size="sm" />
7173

72-
<img src={cp_remove} alt="Remove notice" />
74+
<Image img={cp_remove} alt="Remove notice" size="lg" />
7375

7476
The summary table provides controls to display sample data from the source or the destination table in ClickHouse
7577

76-
<img src={cp_destination} alt="View destination" />
78+
<Image img={cp_destination} alt="View destination" size="lg" />
7779

7880
As well as controls to remove the ClickPipe and display a summary of the ingest job.
7981

80-
<img src={cp_overview} alt="View overview" />
82+
<Image img={cp_overview} alt="View overview" size="lg" />
8183

8284
10. **Congratulations!** you have successfully set up your first ClickPipe. If this is a streaming ClickPipe it will be continuously running, ingesting data in real-time from your remote data source. Otherwise it will ingest the batch and complete.
8385

docs/integrations/data-ingestion/clickpipes/postgres/index.md

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import ssh_tunnel from '@site/static/images/integrations/data-ingestion/clickpip
1414
import select_replication_slot from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/select-replication-slot.jpg'
1515
import select_destination_db from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/select-destination-db.jpg'
1616
import ch_permissions from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/ch-permissions.jpg'
17+
import Image from '@theme/IdealImage';
1718

1819
# Ingesting Data from Postgres to ClickHouse (using CDC)
1920

@@ -61,15 +62,15 @@ Make sure you are logged in to your ClickHouse Cloud account. If you don't have
6162
[//]: # ( TODO update image here)
6263
1. In the ClickHouse Cloud Console, navigate to your ClickHouse Cloud Service.
6364

64-
<img src={cp_service} alt="ClickPipes service" />
65+
<Image img={cp_service} alt="ClickPipes service" size="lg"/>
6566

6667
2. Select the `Data Sources` button on the left-side menu and click on "Set up a ClickPipe"
6768

68-
<img src={cp_step0} alt="Select imports" />
69+
<Image img={cp_step0} alt="Select imports" size="lg"/>
6970

7071
3. Select the `Postgres CDC` tile
7172

72-
<img src={postgres_tile} alt="Select Postgres" />
73+
<Image img={postgres_tile} alt="Select Postgres" size="lg"/>
7374

7475
### Adding your source Postgres database connection {#adding-your-source-postgres-database-connection}
7576

@@ -82,7 +83,7 @@ Make sure you are logged in to your ClickHouse Cloud account. If you don't have
8283

8384
:::
8485

85-
<img src={postgres_connection_details} alt="Fill in connection details" />
86+
<Image img={postgres_connection_details} alt="Fill in connection details" size="lg"/>
8687

8788
#### (Optional) Setting up SSH Tunneling {#optional-setting-up-ssh-tunneling}
8889

@@ -92,7 +93,7 @@ You can specify SSH tunneling details if your source Postgres database is not pu
9293
1. Enable the "Use SSH Tunnelling" toggle.
9394
2. Fill in the SSH connection details.
9495

95-
<img src={ssh_tunnel} alt="SSH tunneling" />
96+
<Image img={ssh_tunnel} alt="SSH tunneling" size="lg"/>
9697

9798
3. To use Key-based authentication, click on "Revoke and generate key pair" to generate a new key pair and copy the generated public key to your SSH server under `~/.ssh/authorized_keys`.
9899
4. Click on "Verify Connection" to verify the connection.
@@ -109,7 +110,7 @@ Once the connection details are filled in, click on "Next".
109110

110111
5. Make sure to select the replication slot from the dropdown list you created in the prerequisites step.
111112

112-
<img src={select_replication_slot} alt="Select replication slot" />
113+
<Image img={select_replication_slot} alt="Select replication slot" size="lg"/>
113114

114115
#### Advanced Settings {#advanced-settings}
115116

@@ -126,21 +127,21 @@ You can configure the Advanced settings if needed. A brief description of each s
126127

127128
6. Here you can select the destination database for your ClickPipe. You can either select an existing database or create a new one.
128129

129-
<img src={select_destination_db} alt="Select destination database" />
130-
130+
<Image img={select_destination_db} alt="Select destination database" size="lg"/>
131+
131132
7. You can select the tables you want to replicate from the source Postgres database. While selecting the tables, you can also choose to rename the tables in the destination ClickHouse database as well as exclude specific columns.
132133

133134
:::warning
134135

135136
If you are defining a Ordering Key in ClickHouse differently from the Primary Key in Postgres, please don’t forget to read all the [considerations](https://docs.peerdb.io/mirror/ordering-key-different) around it!
136-
137+
137138
:::
138139

139140
### Review permissions and start the ClickPipe {#review-permissions-and-start-the-clickpipe}
140141

141142
8. Select the "Full access" role from the permissions dropdown and click "Complete Setup".
142143

143-
<img src={ch_permissions} alt="Review permissions" />
144+
<Image img={ch_permissions} alt="Review permissions" size="lg"/>
144145

145146
## What's next? {#whats-next}
146147

@@ -153,4 +154,3 @@ Also, please refer to the [ClickPipes for Postgres FAQ](./postgres/faq) for more
153154
[This](https://docs.peerdb.io/bestpractices/clickhouse_datamodeling) is especially important as ClickHouse differs from Postgres, and you might encounter some surprises. This guide helps address potential pitfalls and ensures you can take full advantage of ClickHouse.
154155

155156
:::
156-

docs/integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import server_parameters from '@site/static/images/integrations/data-ingestion/c
99
import wal_level from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres/wal_level.png';
1010
import restart from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres/restart.png';
1111
import firewall from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/azure-flexible-server-postgres/firewall.png';
12+
import Image from '@theme/IdealImage';
1213

1314
# Azure Flexible Server for Postgres Source Setup Guide
1415

@@ -20,15 +21,15 @@ ClickPipes supports Postgres version 12 and later.
2021

2122
1. Click on the **Server parameters** section
2223

23-
<img src={server_parameters} alt="Server Parameters in Azure Flexible Server for Postgres" />
24+
<Image img={server_parameters} alt="Server Parameters in Azure Flexible Server for Postgres" size="lg"/>
2425

2526
2. Edit the `wal_level` to `logical`
2627

27-
<img src={wal_level} alt="Change wal_level to logical in Azure Flexible Server for Postgres" />
28+
<Image img={wal_level} alt="Change wal_level to logical in Azure Flexible Server for Postgres" size="lg"/>
2829

2930
3. This change would require a server restart. So restart when requested.
3031

31-
<img src={restart} alt="Restart server after changing wal_level" />
32+
<Image img={restart} alt="Restart server after changing wal_level" size="lg"/>
3233

3334
## Creating ClickPipes User and Granting permissions {#creating-clickpipes-user-and-granting-permissions}
3435

@@ -74,7 +75,7 @@ Please follow the below steps to add [ClickPipes IPs](../../index.md#list-of-sta
7475
1. Go to the **Networking** tab and add the [ClickPipes IPs](../../index.md#list-of-static-ips) to the Firewall
7576
of your Azure Flexible Server Postgres OR the Jump Server/Bastion if you are using SSH tunneling.
7677

77-
<img src={firewall} alt="Add ClickPipes IPs to Firewall in Azure Flexible Server for Postgres" />
78+
<Image img={firewall} alt="Add ClickPipes IPs to Firewall in Azure Flexible Server for Postgres" size="lg"/>
7879

7980

8081
## What's next? {#whats-next}

docs/integrations/data-ingestion/clickpipes/postgres/source/crunchy-postgres.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ title: 'Crunchy Bridge Postgres Source Setup Guide'
77

88
import firewall_rules_crunchy_bridge from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/crunchy-postgres/firewall_rules_crunchy_bridge.png'
99
import add_firewall_rules_crunchy_bridge from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/crunchy-postgres/add_firewall_rules_crunchy_bridge.png'
10+
import Image from '@theme/IdealImage';
1011

1112
# Crunchy Bridge Postgres Source Setup Guide
1213

@@ -57,9 +58,9 @@ Connect to your Crunchy Bridge Postgres through the `postgres` user and run the
5758
5859
Safelist [ClickPipes IPs](../../index.md#list-of-static-ips) by adding the Firewall Rules in Crunchy Bridge.
5960
60-
<img src={firewall_rules_crunchy_bridge} alt="Where to find Firewall Rules in Crunchy Bridge?"/>
61+
<Image size="lg" img={firewall_rules_crunchy_bridge} alt="Where to find Firewall Rules in Crunchy Bridge?"/>
6162
62-
<img src={add_firewall_rules_crunchy_bridge} alt="Add the Firewall Rules for ClickPipes"/>
63+
<Image size="lg" img={add_firewall_rules_crunchy_bridge} alt="Add the Firewall Rules for ClickPipes"/>
6364
6465
6566

docs/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql.md

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import connections from '@site/static/images/integrations/data-ingestion/clickpi
1313
import connections_networking from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql/connections_networking.png';
1414
import firewall1 from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql/firewall1.png';
1515
import firewall2 from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/google-cloudsql/firewall2.png';
16+
import Image from '@theme/IdealImage';
1617

1718
# Google Cloud SQL Postgres Source Setup Guide
1819

@@ -33,13 +34,13 @@ Anything on or after Postgres 12
3334

3435
1. Click on **Edit** button on the Overview page.
3536

36-
<img src={edit_button} alt="Edit Button in Cloud SQL Postgres" />
37+
<Image img={edit_button} alt="Edit Button in Cloud SQL Postgres" size="lg"/>
3738

3839
2. Go to Flags and change `cloudsql.logical_decoding` to on and `wal_sender_timeout` to 0. These changes will need restarting your Postgres server.
3940

40-
<img src={cloudsql_logical_decoding1} alt="Change cloudsql.logical_decoding to on" />
41-
<img src={cloudsql_logical_decoding2} alt="Changed cloudsql.logical_decoding and wal_sender_timeout" />
42-
<img src={cloudsql_logical_decoding3} alt="Restart Server" />
41+
<Image img={cloudsql_logical_decoding1} alt="Change cloudsql.logical_decoding to on" size="lg"/>
42+
<Image img={cloudsql_logical_decoding2} alt="Changed cloudsql.logical_decoding and wal_sender_timeout" size="lg"/>
43+
<Image img={cloudsql_logical_decoding3} alt="Restart Server" size="lg"/>
4344

4445

4546
## Creating ClickPipes User and Granting permissions {#creating-clickpipes-user-and-granting-permissions}
@@ -87,16 +88,16 @@ If your are using SSH Tunneling, then you need to add the [ClickPipes IPs](../..
8788

8889
1. Go to **Connections** section
8990

90-
<img src={connections} alt="Connections Section in Cloud SQL" />
91+
<Image img={connections} alt="Connections Section in Cloud SQL" size="lg"/>
9192

9293
2. Go to the Networking subsection
9394

94-
<img src={connections_networking} alt="Networking Subsection in Cloud SQL" />
95+
<Image img={connections_networking} alt="Networking Subsection in Cloud SQL" size="lg"/>
9596

9697
3. Add the [public IPs of ClickPipes](../../index.md#list-of-static-ips)
9798

98-
<img src={firewall1} alt="Add ClickPipes Networks to Firewall" />
99-
<img src={firewall2} alt="ClickPipes Networks Added to Firewall" />
99+
<Image img={firewall1} alt="Add ClickPipes Networks to Firewall" size="lg"/>
100+
<Image img={firewall2} alt="ClickPipes Networks Added to Firewall" size="lg"/>
100101

101102

102103
## What's next? {#whats-next}

docs/integrations/data-ingestion/clickpipes/postgres/source/neon-postgres.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ import neon_enable_replication from '@site/static/images/integrations/data-inges
1010
import neon_enabled_replication from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/neon-postgres/neon-enabled-replication.png'
1111
import neon_ip_allow from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/neon-postgres/neon-ip-allow.png'
1212
import neon_conn_details from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/setup/neon-postgres/neon-conn-details.png'
13+
import Image from '@theme/IdealImage';
1314

1415
# Neon Postgres Source Setup Guide
1516

@@ -37,19 +38,19 @@ Here, we can run the following SQL commands:
3738
CREATE PUBLICATION clickpipes_publication FOR ALL TABLES;
3839
```
3940

40-
<img src={neon_commands} alt="User and publication commands"/>
41+
<Image size="lg" img={neon_commands} alt="User and publication commands"/>
4142

4243
Click on **Run** to have a publication and a user ready.
4344

4445
## Enable Logical Replication {#enable-logical-replication}
4546
In Neon, you can enable logical replication through the UI. This is necessary for ClickPipes's CDC to replicate data.
4647
Head over to the **Settings** tab and then to the **Logical Replication** section.
4748

48-
<img src={neon_enable_replication} alt="Enable logical replication"/>
49+
<Image size="lg" img={neon_enable_replication} alt="Enable logical replication"/>
4950

5051
Click on **Enable** to be all set here. You should see the below success message once you enable it.
5152

52-
<img src={neon_enabled_replication} alt="Logical replication enabled"/>
53+
<Image size="lg" img={neon_enabled_replication} alt="Logical replication enabled"/>
5354

5455
Let's verify the below settings in your Neon Postgres instance:
5556
```sql
@@ -62,14 +63,14 @@ SHOW max_replication_slots; -- should be 10
6263
If you have Neon Enterprise plan, you can whitelist the [ClickPipes IPs](../../index.md#list-of-static-ips) to allow replication from ClickPipes to your Neon Postgres instance.
6364
To do this you can click on the **Settings** tab and go to the **IP Allow** section.
6465

65-
<img src={neon_ip_allow} alt="Allow IPs screen"/>
66+
<Image size="lg" img={neon_ip_allow} alt="Allow IPs screen"/>
6667

6768
## Copy Connection Details {#copy-connection-details}
6869
Now that we have the user, publication ready and replication enabled, we can copy the connection details to create a new ClickPipe.
6970
Head over to the **Dashboard** and at the text box where it shows the connection string,
7071
change the view to **Parameters Only**. We will need these parameters for our next step.
7172

72-
<img src={neon_conn_details} alt="Connection details"/>
73+
<Image size="lg" img={neon_conn_details} alt="Connection details"/>
7374

7475
## What's next? {#whats-next}
7576

docs/integrations/data-ingestion/clickpipes/postgres/source/rds.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ import modify_parameter_group from '@site/static/images/integrations/data-ingest
1212
import reboot_rds from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/reboot_rds.png';
1313
import security_group_in_rds_postgres from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/security_group_in_rds_postgres.png';
1414
import edit_inbound_rules from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/edit_inbound_rules.png';
15+
import Image from '@theme/IdealImage';
1516

1617
# RDS Postgres Source Setup Guide
1718

@@ -47,19 +48,19 @@ If not already configured, follow these steps:
4748
- Set `rds.logical_replication` to 1
4849
- Set `wal_sender_timeout` to 0
4950

50-
<img src={parameter_group_in_blade} alt="Where to find Parameter groups in RDS?" />
51+
<Image img={parameter_group_in_blade} alt="Where to find Parameter groups in RDS?" size="lg" />
5152

52-
<img src={change_rds_logical_replication} alt="Changing rds.logical_replication" />
53+
<Image img={change_rds_logical_replication} alt="Changing rds.logical_replication" size="lg"/>
5354

54-
<img src={change_wal_sender_timeout} alt="Changing wal_sender_timeout" />
55+
<Image img={change_wal_sender_timeout} alt="Changing wal_sender_timeout" size="lg"/>
5556

5657
2. Apply the new parameter group to your RDS Postgres database
5758

58-
<img src={modify_parameter_group} alt="Modifying RDS Postgres with new parameter group" />
59+
<Image img={modify_parameter_group} alt="Modifying RDS Postgres with new parameter group" size="lg"/>
5960

6061
3. Reboot your RDS instance to apply the changes
6162

62-
<img src={reboot_rds} alt="Reboot RDS Postgres" />
63+
<Image img={reboot_rds} alt="Reboot RDS Postgres" size="lg"/>
6364

6465
## Configure Database User {#configure-database-user}
6566

@@ -98,9 +99,9 @@ Connect to your RDS Postgres instance as an admin user and execute the following
9899

99100
If you want to restrict traffic to your RDS instance, please add the [documented static NAT IPs](../../index.md#list-of-static-ips) to the `Inbound rules` of your RDS security group.
100101

101-
<img src={security_group_in_rds_postgres} alt="Where to find security group in RDS Postgres?" />
102+
<Image img={security_group_in_rds_postgres} alt="Where to find security group in RDS Postgres?" size="lg"/>
102103

103-
<img src={edit_inbound_rules} alt="Edit inbound rules for the above security group" />
104+
<Image img={edit_inbound_rules} alt="Edit inbound rules for the above security group" size="lg"/>
104105

105106
### Private Access via AWS PrivateLink {#private-access-via-aws-privatelink}
106107

0 commit comments

Comments
 (0)