Bodo

Bodo is a JIT compiler for Pandas/Numpy code that statically types the code during compilation time. However, it infers types automatically (looks at file metadata etc) and not really built for manually typing everything. https://github.com/bodo-ai/Bodo (I work on Bodo)

This is very interesting, clearly there's a major pain point here to be addressed, especially the delta between local pandas work and distributed [pyspark] work! Would love to test this out and do benchmarks against us/ Dask/ Spark/ Ray etc which have been our primary testing ground. Full disclosure, work at Bodo which has similar-ish aspirations (https://github.com/bodo-ai/Bodo), but FOSS all the way.

I think staying in Python is best to ensure the team's productivity. There are a lot of tools that help accelerate and scale Pandas code while trying to be as compatible as possible. You can look at "Pandas on Spark", Snowpark (if on Snowflake), Bodo, Dask, and Modin for example. Disclaimer: I work on Bodo and wanted to share it in case others find it useful. https://github.com/bodo-ai/Bodo

Hello HN, I’m excited to share Bodo, an open-source compute engine designed for large-scale data processing in native Python. Bodo is powered by an auto-parallelizing JIT compiler and an HPC backend, enabling it to generate highly optimized, parallel binaries (MPI) for Pandas and NumPy code—all without requiring any code rewrites. Our latest benchmark demonstrates 20x to 240x speedup over traditional distributed computing frameworks like Spark, Ray, and Dask (code and details in repo). The inspiration for Bodo came from my background in HPC, when I saw how extremely slow and hard to use Spark was (has gotten better over the years but still not great). Of course, a compiler has its own limitations (e.g. not all Python is compilable), but I think it’s leaps and bounds better. Let me know what you think.