Skip to content

Commit c1b5b0b

Browse files
authored
[clickpipes] Add recommendation for max_slot_wal_keep_size (#3180)
1 parent 6751aee commit c1b5b0b

File tree

1 file changed

+36
-0
lines changed
  • docs/en/integrations/data-ingestion/clickpipes/postgres

1 file changed

+36
-0
lines changed

docs/en/integrations/data-ingestion/clickpipes/postgres/faq.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,3 +147,39 @@ These adjustments should significantly enhance the performance of the initial lo
147147
### How should I scope my publications when setting up replication?
148148
149149
You can let ClickPipes manage your publications (requires write access) or create them yourself. With ClickPipe-managed publications, we automatically handle table additions and removals as you edit the pipe. If self-managing, carefully scope your publications to only include tables you need to replicate - including unnecessary tables will slow down Postgres WAL decoding. Importantly, exclude tables without primary keys if you're not replicating them to avoid potential replication slowness.
150+
151+
152+
## Recommended `max_slot_wal_keep_size` Settings
153+
154+
- **At Minimum:** Set [`max_slot_wal_keep_size`](https://www.postgresql.org/docs/devel/runtime-config-replication.html#GUC-MAX-SLOT-WAL-KEEP-SIZE) to retain at least **two days' worth** of WAL data.
155+
- **For Large Databases (High Transaction Volume):** Retain at least **2-3 times** the peak WAL generation per day.
156+
- **For Storage-Constrained Environments:** Tune this conservatively to **avoid disk exhaustion** while ensuring replication stability.
157+
158+
### How to Calculate the Right Value
159+
160+
To determine the right setting, measure the WAL generation rate:
161+
162+
#### For PostgreSQL 10+:
163+
164+
```sql
165+
SELECT pg_wal_lsn_diff(pg_current_wal_insert_lsn(), '0/0') / 1024 / 1024 AS wal_generated_mb;
166+
```
167+
168+
#### For PostgreSQL 9.6 and below:
169+
170+
```sql
171+
SELECT pg_xlog_location_diff(pg_current_xlog_insert_location(), '0/0') / 1024 / 1024 AS wal_generated_mb;
172+
```
173+
174+
* Run the above query at different times of the day, especially during highly transactional periods.
175+
* Calculate how much WAL is generated per 24-hour period.
176+
* Multiply that number by 2 or 3 to provide sufficient retention.
177+
* Set `max_slot_wal_keep_size` to the resulting value in MB or GB.
178+
179+
#### Example:
180+
181+
If your database generates 100 GB of WAL per day, set:
182+
183+
```sql
184+
max_slot_wal_keep_size = 200GB
185+
```

0 commit comments

Comments
 (0)