weld

https://github.com/weld-project/weld this is in the same domain of vaex I suppose, the scope is much larger though

Have you heard about Weld[1] ? It is an ongoing effort to built an intermediary representation that numerical library could share to produce an optimized instructions tree. [1] https://github.com/weld-project/weld

It's very easy to get people to pay for hosting. In Mozilla's case, I would love to see some sort of cloud hosting for Rust. I also wish I could write front end in Rust but maybe I'll just have to wait for webassembly. I can't imagine that some sort of rust to js compiler (kinda like kotlin) would be too hard and this would be a killer application. Matei Zaharia (of the Spark fame) and friends are working on weld, https://github.com/weld-project/weld. There's going to be a lot of money there as well. A cloudera competitor of sorts. But yeah, go ham on Rust based paid solution.

A deeper link outlining the Weld-pandas integration https://github.com/weld-project/weld/blob/master/python/griz...

I didn't quite get it from the initial skim of this page. But here's the key bit: > [existing solutions slow] ... due to extensive data movement across the functions. Weld’s take on solving this problem is to lazily build up a computation for the entire workflow, optimizing and evaluating it only when a result is needed. The "Background" [1] section of the tutorial has more info too. I've done lots of optimizations before with CPU/GPU computations and I've definitely noticed that minimizing trips across the bus and doing computation during bus activity is critical to saturating the processors/memory. But would we see the orders-of-magnitude improvements shown on this page for CPU-only work? I can't quite tell what was measured in the graphs. [1] https://github.com/weld-project/weld/blob/master/docs/tutori...