TOtomnipotent·Dec 29, 2016
>> pushing data frames in and out of memory as simple as possible
Where do you think its pushing from and to? Parquet.
Wes McKinney (of pandas fame) is the author behind Arrow/Feather as well as parquet-cpp [1]. Arrow is a compact binary representation of columnar data for in-memory processing with the HUGE benefit that data can be copied over the network with no serialization/deserialization between processes or languages.
One of the primary advantages of Spark over MapReduce is that it doesn't require data to be copied to disk and incur costly SerDer overhead. Now imagine that not only can I avoid writing to disk between processing stages, I can now move GBs of data between nodes by simply copying data from the network stack to memory as-is.
>> Parquet is fantastic, but it has a lot of functionality that's not very relevant for sub-TB o
Read the rest
f data, local use.
Parquet files can still be on local storage, there's no dependency on HDFS or network storage. If I'm only dealing with MBs then CSV will do the trick, otherwise Parquet wins in every category when reading/writing data.
[1] https://github.com/apache/parquet-cpp