sgr

It’s hard to say without hearing more details, but we might be able to help you at Splitgraph. Here’s a recent blog post [0] about using GitHub Actions to orchestrate data repositories in Splitgraph Cloud (SaaS, public site is free and private deployments cost money). And here’s an example [1] of using local Splitgraph (open source, no dependency on any SaaS) to switch between versions of data while querying it in a Jupyter notebook. If you’d like to share more details about your use case, feel free to send me an email (see profile). The product is a bit “all over the place” right now, so I’d be happy to show you specifically how you could use it to solve your problem. [0] https://www.splitgraph.com/blog/splitgraph-yml-terraform-dat... [1] https://github.com/splitgraph/splitgraph/tree/master/example...

You might like what we're doing with Splitgraph. Our command line tool (sgr) installs an audit log into Postgres to track changes [0]. Then `sgr commit` can write these changes to delta-compressed objects [1], where each object is a columnar fragment of data, addressable by the LTHash of rows added/deleted by the fragment, and attached to metadata describing its index [2]. I haven't explored sirix before, but at first glance it looks like we have some similar ideas — thanks for sharing, I'm excited to learn more, especially about its application of ZFS. [0] https://www.splitgraph.com/docs/working-with-data/tracking-c... [1] https://www.splitgraph.com/docs/concepts/objects [2] https://github.com/splitgraph/splitgraph/blob/master/splitgr...