fastparquet

fsspec is used in dask (https://github.com/dask/dask) if you want to see it in action. The author of fsspec also created fastparquet (https://github.com/dask/fastparquet), a native Python implementation of the Parquet file format. I'm really appreciative for Martin's extensive contributions to the PyData ecosystem.

fastparquet https://github.com/dask/fastparquet

So stuff like this or marshmallow is more for cases when you have some database / ORM objects and you want to serialize them out to a json object, or you want to process form/POST data into a well-structured json or database object. For your use case, it's more about large amounts of tabular data and efficient (binary / columnar / compressed) serialization and queryability. I'd say that the defacto standard for that is the HDF5 standard, which PyTables supports (http://www.pytables.org/). This is what pandas uses under the hood and I've been using this with hundreds of millions of rows with no problem. Arrow is slightly more different - it's a specification for the in-memory layout of data that enables faster computation. This is more about what happens if you have data in memory and you want to use it with another tool - serializing / de

Your link is malformed. https://github.com/dask/fastparquet

Are you familiar with fastparquet (https://github.com/dask/fastparquet) and pyarrow (https://pyarrow.readthedocs.io/en/latest) ?