fletcher

Arrow + Parquet is brilliant! Right now I'm writing tools in Python (Python!) to analyse several 100TB datasets in S3. Each dataset is made up of 1000+ 6GB parquet files (tables UNLOADed from AWS Redshift db). Parquet's columnar compression gives a 15x reduction in on-disk size. Parquet also stores chunk metadata at the end of each file, allowing reads to skip over most data that isn't relevant. And once in memory, the Arrow format gives zero-copy compatibility with Numpy and Pandas. If you try this with Python, make sure you use the latest 2.0.0 version of PyArrow [1]. Two other interesting libraries for manipulating PyArrow Tables and ChunkedArrays are fletcher [2] and graphique[3]. [1] I use: conda install -c conda-forge pyarrow python-snappy [2] https://github.com/xhochy/fletcher [3] https://github.com/coady/graphique

I just did one using fletcher (https://github.com/xhochy/fletcher), a library that extends pandas arrays with arrow arrays: n = 2**25 data = np.random.choice(list(string.ascii_letters), n) df = pd.DataFrame({ 'string': data, 'arrow': fl.FletcherArray(data), 'categorical': pd.Categorical(data), 'ints': np.arange(n) }) for a groupby operation on string/arrow/categorical and sum the ints the results are: - String: 1.58 s - Arrow: 886 ms - Categorical: 406 ms The base type of this string FletcherArray array is <pyarrow.lib.ChunkedArray, so maybe there is another more convenient arrow array type or the bottleneck is in upper layer around the groupby operation (since it only is twice as fast). Any way, for this kind of operation the Categorical is the winner and double speed is quite a no