bottledwater-pg

FYI I've used bottledwater and it can cause segfaults that take down the postgres database hard. I ran into both of these: https://github.com/confluentinc/bottledwater-pg/issues/53 https://github.com/confluentinc/bottledwater-pg/issues/61 They're fixed now, but this is very much beta stuff. It wasn't as magical as we'd hoped so we ended up ripping it out and switching the application to use SSE[1] which streamed into a kafka producer. [1] https://en.wikipedia.org/wiki/Server-sent_events

Postgres 9.4+ lets you use the write-ahead log and "replication slots" using which you can create streams of database changes that can be replayed to a client in the order they were made on the origin server [1] EDIT: In case the question was which library did I use, I used BottledWater and it's worked great for me so far [2] [1] https://www.postgresql.org/docs/9.4/static/logicaldecoding-e... [2] https://github.com/confluentinc/bottledwater-pg

What is the best way to ETL data from one Postgres cluster to another (by ETL I mean things like data sanitization and denormalization)? a) I think most people end up doing some kind of batch synchronization, but I'm interested in streaming solutions. b) A lot of folks use trigger based replication, but triggers have to be on the primary/master node, and not just on the replicas. c) Another common solution is to force the database client to write to a message broker, but that opens the door to data discrepancies and synchronization issues. d) In theory I think the best way is to do something like bottledwater-pg or pg_kafka [1] [2] [3], but I'm not sure how battle hardened these are. I think logical replication of the WAL is the right approach, but there is still not much tooling around this. [1] https://github.com/confluentinc/bottledwa