delta-rs

If I understand correctly, polars relies on delta-rs for Delta Lake support, and that is what does not support Deletion vectors: https://github.com/delta-io/delta-rs/issues/1094 It seems like these single-node libraries can process a terabyte on a typical machine, and you'd have have over 10TB before moving to Spark.

You can already have it in Delta with Delta Rust and Python bindings: https://github.com/delta-io/delta-rs

I work at Databricks, but am pretty much just an OSS nerd, mainly focusing on Delta Rust recently: https://github.com/delta-io/delta-rs I did some keyword research and wrote this post cause lots of folks are doing searches for Delta Lake vs Parquet. I'm just trying to share a fair summary of the tradeoffs with folks who are doing this search. It's a popular post and that's why I figured I would share it here.

PyO3 is being used to expose the Python bindings to the delta-rs project: https://github.com/delta-io/delta-rs It's a great way to expose Python bindings because it "feels" Pythonic. Most users run pip install deltalake and are completely unaware that the backend is implemented in Rust. This is quite a different user experience than Python bindings for Java backends exposed via py4j. The py4j interfaces have the Java feel and require Java to be installed, which most Python users don't like.

> You cannot browse the data structure without an active cluster That's one of the reason's I'm interested in delta-rs [1], which has delta lake bindings for Python. Would love to read a delta lake table into a native python object without the need for spark. [1] https://github.com/delta-io/delta-rs

This looks to be the integration test: https://github.com/delta-io/delta-rs/commit/09d345e52ca3177d... I'm not sure why the author thinks this required putting the credentials in the code itself. Heck, I'd argue that this integration test should be setting up the bucket and contents as part of the setup for the test itself, pulling the credentials from some sort of secrets manager.