Skip to content

Commit f9fa2d6

Browse files
committed
FAQ for Postgres slot memalloc bug
1 parent 4864cde commit f9fa2d6

File tree

2 files changed

+13
-3
lines changed

2 files changed

+13
-3
lines changed

docs/integrations/data-ingestion/clickpipes/postgres/faq.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,11 @@ You have two options when dealing with tables without primary keys:
183183
If you're creating a publication manually instead of letting ClickPipes manage it, we don't recommend creating a publication `FOR ALL TABLES`, this leads to more traffic from Postgres to ClickPipes (to sending changes for other tables not in the pipe) and reduces overall efficiency.
184184
185185
For manually created publications, please add any tables you want to the publication before adding them to the pipe.
186-
:::
186+
:::
187+
188+
:::warning
189+
If you're replicating from a Postgres read replica/hot standby, you will need to create your own publication on the primary instance, which will automatically propagate to the standby. The ClickPipe will not be able to manage the publication in this case as you're unable to create publications on a standby.
190+
:::
187191
188192
## Recommended `max_slot_wal_keep_size` settings {#recommended-max_slot_wal_keep_size-settings}
189193
@@ -272,9 +276,13 @@ WITH (publish_via_partition_root = true);
272276
This error typically occurs when the source Postgres database has a datatype which cannot be mapped during ingestion.
273277
For more specific issue, refer to the possibilities below.
274278

275-
### `Cannot parse type Decimal(XX, YY), expected non-empty binary data with size equal to or less than ...` {#cannot-parse-type-decimal-expected-non-empty-binary-data-with-size-equal-to-or-less-than}
279+
## `Cannot parse type Decimal(XX, YY), expected non-empty binary data with size equal to or less than ...` {#cannot-parse-type-decimal-expected-non-empty-binary-data-with-size-equal-to-or-less-than}
276280

277281
Postgres `NUMERIC`s have really high precision (up to 131072 digits before the decimal point; up to 16383 digits after the decimal point) and ClickHouse Decimal type allows maximum of (76 digits, 39 scale).
278282
The system assumes that _usually_ the size would not get that high and does an optimistic cast for the same as source table can have large number of rows or the row can come in during the CDC phase.
279283

280284
The current workaround would be to map the NUMERIC type to string on ClickHouse. To enable this please raise a ticket with the support team and this will be enabled for your ClickPipes.
285+
286+
## I'm seeing errors like `invalid memory alloc request size <XXX>` during replication/slot creation {#postgres-invalid-memalloc-bug}
287+
288+
There was a bug introduced in Postgres patch versions 17.5/16.9/15.13/14.18/13.21 due to which certain workloads can cause an exponential increase in memory usage, leading to a memory allocation request >1GB which Postgres considers invalid. This bug [has been fixed](https://github.com/postgres/postgres/commit/d87d07b7ad3b782cb74566cd771ecdb2823adf6a) and will be in the next Postgres patch series (17.6...). Please check with your Postgres provider when this patch version will be available for upgrade. If an upgrade isn't immediately possible, a resync of the pipe will be needed as it hits the error.

docs/integrations/data-ingestion/clickpipes/postgres/source/timescale.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,11 +83,13 @@ Make sure to replace `clickpipes_user` and `clickpipes_password` with your desir
8383

8484
```sql
8585
-- When adding new tables to the ClickPipe, you'll need to add them to the publication as well manually.
86-
CREATE PUBLICATION clickpipes_publication FOR TABLE <...>, TABLES IN SCHEMA _timescaledb_internal;
86+
CREATE PUBLICATION clickpipes_publication FOR TABLE <...>, <...>, TABLES IN SCHEMA _timescaledb_internal;
8787
```
8888

8989
:::tip
9090
We don't recommend creating a publication `FOR ALL TABLES`, this leads to more traffic from Postgres to ClickPipes (to sending changes for other tables not in the pipe) and reduces overall efficiency.
91+
92+
For manually created publications, please add any tables you want to the publication before adding them to the pipe.
9193
:::
9294

9395
:::info

0 commit comments

Comments
 (0)