polars

Polars does have an asof join so I guess the mapping hasn't been made yet from SQL. Could you open up an issue at https://github.com/pola-rs/polars/issues ?

It does not use datafusion. https://github.com/pola-rs/polars/issues/6197

High performance polars dataFrame processing library [0] is written in Rust with Python bindings via PyO3/maturin. It exposes huge dataframes to python and successfully solved the interface boundayr problems described in the article. [0] https://github.com/pola-rs/polars

I built a Python CLI for Parquet interoperability. It runs a given Parquet file through selected readers and shows where it fails with a command. Also uses Hypothesis for fuzzing. Generates tables, runs through a matrix of engines and minimizes the found failures. You can also create a schema beforehand and fuzz within it. I believe it can be useful for Parquet engine maintainers and people with multiple Parquet engines involved in their pipeline. Already found and filed two issues with it, one from scanning Apache's own corpus and the other from fuzzing: - https://github.com/pola-rs/polars/issues/28787 - https://github.com/duckdb/duckdb/issues/24731 repo: https://github.com/sovsparrow/parquity

Here are a few examples you might find credible: - pola-rs/polars: https://github.com/pola-rs/polars/pull/26823 - ~2.68x median speedup of primitive-to-boolean casting credited to Claude Opus 4.6 - pydantic/monty: https://github.com/pydantic/monty/pull/643 - ~53x speedup (488ms down to 9.2ms) of bytes substring search generated with Claude Code - numpy/numpy: https://github.com/numpy/numpy/pull/31573 - ~21x speedup of Python datetime → datetime64 conversion, with Claude Code used for profiling and implementing the performance improvements - apache/datafusion: https://github.com/apache/datafusion/pull/21182 - up to 49x speedup of LIMIT queries by eliminating unnecessary sorts and pushing limits into file scans, generated with Claude Code - apache/datafusion: https://github.com/apache/datafusion/pull/21651 - ~4.39x speedup of ClickBench Q6

Would there be an advantage to reusing SIMD-optimized libraries for this? Polars Rust is built on Arrow and packed_simd. pola-rs/polars: https://github.com/pola-rs/polars polars - Rust API docs: https://docs.pola.rs/api/rust/dev/polars/#simd lancedb's data format; Lancedb/lance works with [Pandas, DuckDB, Polars, PyArrow,]; https://github.com/lancedb/lance Narwhals' df interface (Python) https://narwhals-dev.github.io/narwhals/ substrait's portable query plans: https://substrait.io/ , https://github.com/ibis-project/ibis-substrait Arrow RecordBatch, https://news.ycombinator.com/item?id=45495738#45546244 cargo-fuzz, TLA+

`row_index()` was also recently added. df.with_columns(pl.col.b + pl.row_index().is_between(1, 3)) # shape: (5, 2) # ┌─────┬─────┐ # │ a ┆ b │ # │ --- ┆ --- │ # │ i64 ┆ i64 │ # ╞═════╪═════╡ # │ 1 ┆ 10 │ # │ 2 ┆ 21 │ # │ 3 ┆ 31 │ # │ 4 ┆ 41 │ # │ 5 ┆ 50 │ # └─────┴─────┘ > Polars has an optimization to overwite a single value I believe it is just "syntax sugar" for calling `Series.scatter()`[1] > it doesn't allow slicing I believe you are correct: df_polars[1:3, "b"] += 1 # TypeError: cannot use "slice(1, 3, None)" for indexing You can do: df_polars[list(range(1, 4)), "b"] += 1 Perhaps nobody has requested slice syntax? It seems like it would be easy to add. [1]: https://github.com/pola-rs/polars/blob/9079e20ae59f8c75dcce8...

> Ruby and Python are written mostly in C underneath. > > Many try to replace C. All failed. What do you mean by all failed? Rust supports CPython modules using PyO3 [1]. Even the popular polars dataframe library [2] uses it. [1] https://github.com/PyO3/pyo3 [2] https://github.com/pola-rs/polars

Agreed that the polars interface is far superior to SQL! There are a few ways to do this if there's interest...polars wasn't an option because we needed Arrow extension types (https://github.com/pola-rs/polars/issues/9112).

Thanks for creating this issue, it is worth investigating! I see you also created similar issues in Polars: https://github.com/pola-rs/polars/issues/17932 and DuckDB: https://github.com/duckdb/duckdb/issues/17066 ClickHouse has a built-in memory tracker, so even if there is not enough memory, it will stop the query and send an exception to the client, instead of crashing. It also allows fair sharing of memory between different workloads. You need to provide more info on the issue for reproduction, e.g., how to fill the tables. 16 GB of memory should be enough even for a CROSS JOIN between a 10 billion-row and a 100-row table, because it is processed in a streaming fashion without accumulating a large amount of data in memory. The same should be true for a merge join. However, there are places when a large buffer might be needed. For examp

This is false. The polars api has used smart string for a long time. https://github.com/pola-rs/polars/blob/32a2325b55f9bce81d019...

Yes only found the announcement [1] that the Polars team and NVIDIA engineers are working on a GPU engine, but other than that no concrete examples. Github issues also don't provide any hints on the status, only one open item [2] where most comments are prior to the announcement. [1] https://pola.rs/posts/polars-on-gpu/ [2] https://github.com/pola-rs/polars/issues/13111