feather

Also Feather: https://github.com/wesm/feather

It'd be interesting to know how this compares with alternative solutions. I might not understand the benefit proposition correctly, and I'm not specifically into Python for data work, but I immediately thought of things like feather[1], fst[2], disk.frame[3] and even DuckDB[4]. Some of these are on disk rather than in memory, but I'd still be interested in performance and use case comparisons. [1] https://github.com/wesm/feather [2] https://www.fstpackage.org/fst/ [3] https://diskframe.com/ [4] https://duckdb.org/

Is this any different than Parquet (https://parquet.apache.org/ -- index, columnar data split across files), Feather (https://github.com/wesm/feather -- which has first class R support or FST (https://www.fstpackage.org/ -- Also first class R support)?

pretty cool and thanks for that reply! Did you look at something like Arrow/Feather, which is looking to get adopted as the interoperable format in R/Pandas ... and maybe even Spark. There's been quite a bit of momentum behind it to optimize it for huge usecases - https://thenewstack.io/apache-arrow-designed-accelerate-hado... It is based on Google Flatbuffers, but is undergoing enough engineering specifically from a big data/machine learning perspective. Instead of building directly over Protobuf, it might be interesting to build it on top of Arrow (in exactly the same way that Feather is based on top of Arrow https://github.com/wesm/feather).

So stuff like this or marshmallow is more for cases when you have some database / ORM objects and you want to serialize them out to a json object, or you want to process form/POST data into a well-structured json or database object. For your use case, it's more about large amounts of tabular data and efficient (binary / columnar / compressed) serialization and queryability. I'd say that the defacto standard for that is the HDF5 standard, which PyTables supports (http://www.pytables.org/). This is what pandas uses under the hood and I've been using this with hundreds of millions of rows with no problem. Arrow is slightly more different - it's a specification for the in-memory layout of data that enables faster computation. This is more about what happens if you have data in memory and you want to use it with another tool - serializing / de

I guess I never found the <- syntax odd, because that's all I know. Not using = for assignment kind of makes sense to me, as equals has mathematical connotations. I always thought there were plenty of options for parallel and high(er) performance computing with R.[1] Data sets which don't entirely fit in memory can be stored pretty much anywhere and queried. Simplest would be an SQLite database. Formats like Feather[2] and fst[3] are also really useful. There are multiple R interfaces for Redis.[4] R has a library to interact with the Tensorflow API[5] and many other machine learning services. "Multiple legacy object semantics" doesn't really mean a lot to me. Knowing nothing else, that's just the way R is, plus I don't think there are that many really strange ways of doing things. Mostly there's a discernable pattern across functions. So

If you want a more efficient version of pickle, take a look at feather [1] It's not a drop in replacement but it's very effective 1 - https://github.com/wesm/feather

> Wes -- does the Python library for Feather already release the GIL? Checking out the code: https://github.com/wesm/feather/blob/master/python/feather/l... So the actual reading from and writing to disk should be with a released GIL if I'm reading this correctly. The conversion to and from arrays or dataframes holds the GIL.

Is there a spec somewhere, or is the existing implementation the spec? Edit: https://github.com/wesm/feather/blob/master/doc/FORMAT.md Seems a bit sparse/incomplete still (as would be expected for a brand new project).

Feather doesn't support a number of pandas features (but these are features that R doesn't really support, either): https://github.com/wesm/feather/blob/master/python/README.md (not comprehensive)