pg_clickhouse

tbuffers working with lua (it was buggy, opened half a dozen PRs to flatbuffers repo which got ignored). It was clearly awful having to manage flatbuffer schema files & update scripts every time schema changed. But I had alternative already made: msgpack. Throughput needed work but addressed that by creating pool of lua interpreters Overall I overworked myself (put my hands out of commission & spent months relearning how to type on split ergo colemak-dh), but I enjoyed the work. Team was very open with each other & when performance is your selling point there's an understanding that engineering quality needs to be maintained. Sure there were parts of the system I hated, & sometimes I'd try chip away at those Hopefully that helps, hard to say the difference, but I really feel in my work that when customer has problem I'm part of conversation. Most recently there was talk of customer wanting cold data offloaded from postgres which is what inspired https://github.com/ClickHouse/pg_clickhouse/pull/298 where we get Postgres to do most the work Raised problems trying to mix C++ into postgres extension, decided fix was to write clickhouse-c library to replace clickhouse-cpp, there was some doubt on team about value, but demonstrated value (https://github.com/ClickHouse/pg_clickhouse/pull/254) & I appreciate my colleagues not being afraid to change their mind There's a level of trust where instead of being assigning tasks on a board I instead work on what I think is important based on information available. Nobody was asking for wal-rus, but I know my fleet ClickHouse Cloud similarly took route of taking its time hiring sales. Better to have a small sales team that can work directly with engineering on quality leads than overwhelming everyone so that sales becomes the enemy. Guess the difference is agency. When engineering is involved in making commitments they're invested in delivering & there's push back so sales doesn't start hallucinating features

and WAL spikes from long-running transactions are common, usually driven by disk I/O bottlenecks. With fewer I/O constraints, Postgres could handle a lot more. On the analytics side, many limitations come from the fact that Postgres was designed for OLTP and lacks features common in analytical databases, such as vectorized execution and broad ingest format support. As a result, many companies (e.g., GitLab, Ramp, Cloudflare) complement Postgres with ClickHouse to offload analytics, adopting two purpose-built open-source databases. However, adding ClickHouse to a Postgres-based application isn’t straightforward—you typically need to build CDC pipelines, manage backfills and schema changes, and update application code to handle a second database. Solution - NVMe-backed Postgres natively integrated with ClickHouse On the OLTP side, NVMe-based Postgres can dramatically improve performance. By colocating storage with compute, NVMe enables much lower latency and higher IOPS than network-attached storage, speeding up disk-bound operations (up to 10x), including updates, upserts, vacuums, and checkpointing. We’re working on a detailed blog exploring how NVMe reduces WAL fsync, buffer read, and checkpoint overhead, stay tuned! On the OLAP side, the Postgres service offers native CDC to ClickHouse and unified querying via pg_clickhouse. CDC is currently powered by ClickPipes/PeerDB using logical replication, and we’re working to make it faster and simpler with logical replication v2, a new decoding plugin, and a path toward sub-second replication. Every Postgres comes with the pg_clickhouse (https://github.com/ClickHouse/pg_clickhouse) extension, which simplifies adding ClickHouse-powered analytics to a Postgres application. It lets you query ClickHouse directly from Postgres, enabling a single interface for transactions and analytics. pg_clickhouse supports broad analytics query pushdown, with plans to expand this further over time. Who is it for? 1. Companies who are looking for a faster and more scalable Postgres. 2. Companies already using ClickHouse that want a tightly integrated Postgres to offload analytics to ClickHouse, without the operational burden of managing the integration. 3. Companies that are growing rapidly (or expect to) and want a simple yet powerful stack with Postgres on NVMe + ClickHouse. CTA 1. If the above value and vision resonates with you and you want to try it for a use-case, please sign up here and we will give you access within a day or two! https://clickhouse.com/cloud/postgres 2. We’d love to hear your feedback and anything else that comes to mind, it would be super helpful to us as we build this out!

able. We’ll be doubling down to add pushdown support for much more complex queries, CTEs, window functions, and more. More on the future here - https://github.com/ClickHouse/pg_clickhouse?tab=readme-ov-fi... All with the goal of enabling users to build fast analytics from the Postgres layer itself but still using the power of ClickHouse!