CRoaring

HyperLogLog and PostgreSQL: https://github.com/citusdata/postgresql-hll Or even simpler, Roaring Bitmaps: https://pncnmnp.github.io/blogs/roaring-bitmaps.html https://blog.quastor.org/p/grab-rate-limiting https://github.com/RoaringBitmap/CRoaring

The PR in question seems to be https://github.com/RoaringBitmap/CRoaring/pull/446 I agree that the GitHub analyses are worse than useless, but I also think it highlights the friction between intrinsics and the program in which they appear. There probably is a better model.

You mean like https://github.com/RoaringBitmap/CRoaring ?

Though, interestingly, if AVX2 is available then Lemire's CRoaring package uses a Harley-Seal popcount implementation, at https://github.com/RoaringBitmap/CRoaring/blob/master/includ... , rather than the POPCNT64 instruction. Because it's faster - https://arxiv.org/abs/1611.07612 .

How does this compare to https://github.com/RoaringBitmap/CRoaring? Is the difference that this one is distributed?

As the author states in another comment here (https://news.ycombinator.com/item?id=11278640) the charts he has in the article understate the potential advantage of the AVX2 approach over POPCNT. On current processors, POPCNT runs on a single execution port and counts the bits in 4B per cycle. A fast unrolled solution thus gets down very close to .125 cycles/byte (1/8). The AVX2 implementation requires more instructions, but spreads the work out across more ports. My fastest unrolled version takes 2.5 cycles per 32B vector, or .078 cycles/byte (2.5/32). This is 1.6x faster (4 cycles per 32B /2.5 cycles per 32B) than scalar popcnt(). Whether this is worth doing depends on the rest of the workload. If the rest of the work is being done on scalar 64-bit registers, those other scalar operations can often fit between the popcnts() "for fre

However, if the popcounts in your workload remain independent, and you immediately need the values from them, I don't think this approach will help. This can be true, but it depends on whether the rest of the workload is vectorized. If you already have the data in vector registers, or can make use of it there afterward, this approach turns out to be even more beneficial than the microbenchmark numbers imply. We've been working with Wojciech's approach for a while now, and thus can even point to real-world code: https://github.com/RoaringBitmap/CRoaring/blob/master/src/co...