parquet-format

> What does compression do to query performance? That section is the most relevant whenever compression in a DB is discussed. The purpose of a database is to find, aggregate or update data - storage is where the trade-off gets expressed. There are no silver bullets here. Any method of compression which speeds up either filter rejection or scan rate is better than something that only trades off IO for CPU usage. For example, dictionary encoding can be slower to read (because you decompress the whole dictionary and not just the skip read after filter), but not if you can squeeze out an IN clause by turning string comparisons into O(1) dictionary followed by a simple integer filter. Remember, this can be arbitrarily complex (Druid is a great example of this) and then the bitmaps can be used because the dictionary index will be a dense 0-N. E

You can actually make those two bits more independent afaik. https://github.com/apache/parquet-format/blob/master/BloomFi... https://github.com/facebook/rocksdb/blob/main/util/bloom_imp... First one is useful for grasping the idea second one is more comprehensive. Both try to make multiple bit loads but try to make them as independent as possible as far a I can understand. Also hash function has huge effect on bloom filter performance. I was getting 2x perf when using xxhash3 instead of wyhash even though wyhash is a faster hash function afaik.

Parquet files include a field called key_value_metadata in the FileMetadata structure; it sits in the footer of the file. See: https://github.com/apache/parquet-format/blob/master/src/mai... The technique described in the article, seems to use this key-value pair to store pointers to the additional metadata (in this case a distinct index) embedded in the file. Note that we can embed arbitrary binary data in the Parquet file between each data page. This is perfectly valid since all Parquet readers rely on the exact offsets to the data pages specified in the footer. This means that DataFusion does not need to specify how the metadata is interpreted. It is already well specified as part of the Parquet file format itself. DataFusion is an independent project -- it is a query execution engine for OLAP / columnar data, which can take in SQL sta

I've never really worked with parquet or column based systems but just searching for parquet index shows https://github.com/apache/parquet-format/blob/master/PageInd... which describes exactly what I would expect. Maybe it doesn't work well, I don't know- I typically work with structured bundles of data, rather than purely row or column formats (I have a patent on using bundled streams for this purpose).

Right, there's all sorts of metadata and often stats included in any parquet file: https://github.com/apache/parquet-format#file-format The offsets of said metadata are well-defined (i.e. in the footer) so for S3 / blob storage so long as you can efficiently request a range of bytes you can pull the metadata without having to read all the data.

Date is confusing with a timezone (UTC or otherwise) and the doco makes no such suggestion. The Parquet datatypes documentation is pretty clear that there is a flag isAdjustedToUTC to define if the timestamp should be interpreted as having Instant semantics or Local semantics. https://github.com/apache/parquet-format/blob/master/Logical... Still no option to include a TZ offset in the data (so the same datum can be interpreted with both Local and Instant semantics) but not bad really.