universql

primarily with NVME (no use of object stores such as S3 for intermediate data). Thus, it can optimize the workloads much better than the other data warehouses. If you use SQL, another gotcha is that DuckDB doesn't have advanced catalog features in cloud data warehouses. Still, it's possible to combine DuckDB compute and Snowflake Horizon / Databricks Unity Catalog thanks to Apache Iceberg, which enables multi-engine support in the same catalog. I'm experimenting this multi-stack idea with DuckDB <> Snowflake, and it works well so far: https://github.com/buremba/universql

source tables were Iceberg, then you could have an even simpler duckdb file to produce/maintain that would just wrap views around the Iceberg tables using the duckdb iceberg extension. create view x as select * from iceberg_scan('metadata_file'); If you're going to enable change tracking on tables then perhaps another solution is to use CHANGES to select the data from the table instead of reading streams. That way you could use the same timestamp across all the selects and get better consistence between your materializations to duckdb. set ts = {last_materialization_ts}; select * from X CHANGES AT(timestamp=>$ts); select * from Y CHANGES AT(timestamp=>$ts); -- triggered task to kick off materialization to duckdb create task M when SYSTEM$HAS_DATA(X_STREAM) AND SYSTEM$HAS_DATA(Y_STREAM) AND ... AS CALL MY_MATERIALIZE(); -- send SNS, invoke ext-func, etc Here's another similar (but different) project that I've been tracking https://github.com/buremba/universql