streambed

I wonder how many of data infra startups are wrapper on top of S3. Also, the boundary between OLTP and OLAP is blurring every day. For folks who want an off shelf version of this you may be interested in https://github.com/viggy28/streambed (Disclaimer: I am one of the committers)

Boundary between OLTP and OLAP is blurring every day. For folks who want an off shelf version of this you may be interested in https://github.com/viggy28/streambed (Disclaimer: I am one of the committers)

This will welcome more database/warehouses to be written in Go. Personally I will implement it in https://github.com/viggy28/streambed

An analytics store for Postgres on S3 using DuckDB as the query engine https://github.com/viggy28/streambed

Building https://github.com/viggy28/streambed - a full Postgres running on S3 using DuckDB as the query engine.

Tomas Vondra, a major Postgres contributor recently revived a thread on using Bloom filters - https://www.postgresql.org/message-id/flat/5cd8c20c-14b5-4b0... So there is more core work happening on support OLAP but I do think it will take some time. In the meantime, I think we have all the pieces (storage, query engine, table format) to set up a true OLAP. For instance, I created https://github.com/viggy28/streambed to pressure test this idea.

Currently, Strembed expects REPLICA IDENTITY FULL for getting the before and after value of TOAST column. Since we have the data in object storage, we could populate it without the need for REPLICA IDENTITY FULL. Created an issue https://github.com/viggy28/streambed/issues/25 to track this feature.

I wouldn't be surprised, even the the core of the project is heavily vibe-coded[0] [0] https://github.com/viggy28/streambed/blob/a660ebb75b4744f5bd...

Thanks for the kind words! Short answer: yes, column-level schema changes sync to Iceberg automatically[0]. Logical replication (pgoutput in v1) doesn't actually stream DDL statements. Instead, Postgres emits a fresh Relation message describing the table's current column layout right before the next change to that table. So we diff that against the last layout we knew and infer what changed. From there we evolve the Iceberg schema in place: flush any buffered rows under the old schema first, then write a new metadata version with the change. What's handled today: - ADD COLUMN — new field ID allocated; the column's Postgres DEFAULT is carried into Iceberg's initial-default/write-default, so existing rows read back correctly - DROP COLUMN — removed from the current schema, existing data files untouched - Type widening — int4→int8, flo

Are you running self hosted Postgres to run pg_lake? Also, I am working on https://github.com/viggy28/streambed to perform analytics queries on S3 using DuckDB

I was tired of setting up additional read replicas and Snowflake for our BI teams, I built an analytics database using S3 and DuckDB. https://github.com/viggy28/streambed