canceling statement due to conflict with recovery

[ashalitkin]

Tap is frequently used to pull data from the postgres replica and a common issue in this case is "canceling statement due to conflict with recovery" especially for a full table sync. It happens when WAL comes to a replica (upon sync between master and a slave). The tap already has a mechanism to resume but no retry mechanism.
I think it would be helpful to be able to configure number of attempts a tap tries to pull the data.

[ashalitkin]

#114

[Samira-El]

Hey, have you tried bumping max_standby_streaming_delay parameter?

The reason there is no tries to pull the data is that pipelines are usually run with a scheduler, so if they fail, they would try again in the next scheduled run.

[ashalitkin]

Hey, max_standby_streaming_delay affects the replication / master performance and anyway for the long running queries we can still reach the limit.
Very frequently ETL runs e.g. once a day and after extract part is done some other tool transforms the data which will also fail in our case. In other words a manual action will be needed. Why not to do it automatic for this specific case?

[Samira-El]

what makes you think that retrying several times would fix this issue?

Also, I see from the PR you've submitted that you're talking about full table sync, can't you incremental strategy instead?