ALalexisread·Dec 17, 2021
I believe Vaex can do this, in addition to GPU processing and reading direct from s3.
https://github.com/vaexio/vaex
In Polars: Fast DataFrame library for Rust and Pythonvaexio
vaex
Python2018–2021
vaexio / vaex
vaex
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
6Distinct people
10Mentions
0Explicit recommenders
8.5KGitHub stars
What to know before exploring
What it is
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year10 dated mentions
Observed mentions: 2018-12-14 09:30:56 UTC to 2021-12-17 08:53:10 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 4 | 0 | 4 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 3 | 2 | 1 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 3 | 0 | 3 |
What Hacker News says
10 sourcesmaybe give Vaex a try. it's along those lines – out-of-core data frame library, but also works on a single machine
https://github.com/vaexio/vaex
In I wrote one of the fastest DataFrame librariesMy guess is that should be possible, feel free to hop onto https://github.com/vaexio/vaex/discussions !
In A Hybrid Apache Arrow/Numpy DataFrame with Vaex Version 4.0Vaex (Python) = billion rows per second
4 points · Open discussionI've never benchmarked against postgres, but would be interested about the results. I once tried monetdb, and it was orders of magnitude slower for simple calculations, so I stopped looking at RDMS'es after that.
I think they solve a different problem, smaller data, data/relational integrity, data mutations. Dataframes can take shortcuts here.
If you want to do a serious benchmark, feel free to contact us.
Github: https://github.com/vaexio/vaex/issues
Email: (I'm easy to google).
In Show HN: How to analyse 100 GB of data on your laptop with PythonVaex: Out-of-Core DataFrames for Python, visualize and explore big tabular data
1 points · Open discussionThere is an issue open for this: https://github.com/vaexio/vaex/issues/93
It should have been fixed, some more detailed report (version numbers installed) would be good to know.
In Show HN: Vaex - Out of Core Dataframes for Python and Fast VisualizationNote: Vaex has its own memory model. If you input Arrow, it converts to the Vaex data representation. Details here:
https://github.com/vaexio/vaex/blob/master/packages/vaex-arr...
One of the primary objectives of Apache Arrow is to have a common data representation for computational systems, and avoid serialization / conversions altogether.
In Show HN: Vaex - Out of Core Dataframes for Python and Fast VisualizationAbsolutely, I think nowadays the question should be: 'does it still support Python2?' (it does btw)
My question is to you is, would you be so kind to open an issue to decribe the failure on https://github.com/vaexio/vaex/issues ? Please share which OS, which Python distribution (anaconda maybe) and/or the installation steps and error msg.
In Show HN: Vaex - Out of Core Dataframes for Python and Fast VisualizationI have not looked into it, maybe datashader can do this, which is a package purely focussing on viz, while vaex is more allround (although there is overlap).
If you think vaex can be useful here, feel free to ask question/open issues https://github.com/vaexio/vaex
In Show HN: Vaex - Out of Core Dataframes for Python and Fast Visualization