peerdb

You should try PeerDB, it was acquired by ClickHouse for exactly this use-case - Fast, simple Postgres replication to ClickHouse. https://github.com/PeerDB-io/peerdb In ClickHouse Cloud, you have ClickPipes which is a simpler/managed manifesation of PeerDB https://clickhouse.com/cloud/clickpipes/postgres-cdc-connect...

> "As part of the acquisition, we made an important decision: not only to keep PeerDB open source" > "Here goes the Open Source reference to our validation logic." > "PeerDB Open Source Repository" I hate to be that guy, but PeerDB seems to be governed by the Elastic License [1] which makes it NOT open source. The difference is not small, but significant for many. For one, it won't get integrated into other OSS projects. In my particular case, we have integrated Debezium Embedded into StackGres, as a high level object (CRD) named SGStream [2]. It allows Postgres logical replication from Postgres and exports to another Postgres and/or CloudEvents. No Kafka required. We'd love to consider other alternatives like PeerDB, but not being OSS is a red line we can't cross (having said that, we're really happy with Debezium in general, but having

Great question! If you’re starting a greenfield application, pg_clickhouse makes a lot of sense since you’ll be using a unified query layer for your application. Now, coming to your question about replication: you can use PeerDB (acquired by ClickHouse https://github.com/PeerDB-io/peerdb), which is laser-focused and battle-tested at scale for Postgres-to-ClickHouse replication. Once the data is replicated into ClickHouse, you can start querying those tables from within Postgres using pg_clickhouse. In ClickHouse Cloud, we offer ClickPipes for Postgres CDC/replication, which is a managed service version of PeerDB and is tightly integrated with ClickHouse. Now there could be non-transcational tables that you can directly ingest to ClickHouse and still query using pg_clickhouse. So TL;DR: Postgres for OLTP; ClickHouse for OLAP; PeerDB/ClickP

Great question, exactly CDC from Postgres to ClickHouse and adapting the application to start using ClickHouse for analytics. Through the PeerDB acquisition, ClickHouse now has native CDC capabilities that work at any scale (few 10s of GB to 10s of TB Postgres databases). You can use ClickPipes if you’re on ClickHouse Cloud, or PeerDB if you’re using ClickHouse OSS. Sharing a few links for reference: https://clickhouse.com/docs/integrations/clickpipes/postgres https://github.com/PeerDB-io/peerdb https://clickhouse.com/cloud/clickpipes/postgres-cdc-connect... https://clickhouse.com/blog/clickhouse-acquires-peerdb-to-bo... Here is a short demo/talk that we did at our annual conferemce Open House that talks about this reference architecture https://clickhouse.com/videos/postgres-and-clickhouse-the-de...

There seems to be a GH discussion wherein they explain the choice: https://github.com/posit-dev/positron/discussions/3820 If you find that interesting, there are actually quite a few others https://github.com/airbytehq/airbyte-platform/blob/main/LICE... https://github.com/apollographql/federation/blob/main/LICENS... https://github.com/PeerDB-io/peerdb/blob/main/LICENSE.md

I'm guilty of this too https://github.com/PeerDB-io/peerdb/blob/d36da8bb2f4f6c1c821... The inner channel is a poor man's future. Came up with this to have lua runtimes be able to process in parallel while maintaining ordering (A B C in, results of A B C out)