ParallelReductionsBenchmark

I was asked this a few months back but don’t have the measurements fresh anymore. In general, I think TBB is one of the more thorough and feature-rich parallelism libraries out there. That said, I just found a comparable usage example in my benchmarks, and it doesn’t look like TBB will have the same low-latency profile as Fork Union: https://github.com/ashvardanian/ParallelReductionsBenchmark/...

It’s a very interesting benchmark (https://github.com/lemire/TestingMLP) — probably worth adding to the Phoronix or some wider suite. Every couple of years I refresh my own parallel reduction benchmarks (https://github.com/ashvardanian/ParallelReductionsBenchmark), which are also memory-bound. Mine mostly focus on the boring simple throughput-maximizing cases on CPUs and GPUs. Lately, as GPUs are pulled into more general data-processing tasks, I keep running into non-coalesced, pointer-chasing patterns — but I still don’t have a good mental model for estimating the cost of different access strategies. A crossover between these two topics — running MLP-style loads on GPUs — might be exactly the benchmark missing, in case someone is looking for a good weekend project!

For the single threaded version, I believe they have a similar problem with auto sums = _mm256_set1_ps(0); for (; it + 8 < end; it += 8) sums = _mm256_add_ps(_mm256_loadu_ps(it), sums); Where each SMD op is trying to overwrite to a compact data structure. But in the threaded version https://github.com/unum-cloud/ParallelReductions/blob/fd16d9... they have separate slots for an accumulator but it's still in a shared vector, which most likely has the issue I described.

Here is the source and the threads: https://github.com/unum-cloud/ParallelReductions/blob/fd16d9... OFC we don't expect the compiler to instantiate them for us, it's not OpenMP :) That one we covered in previous articles. OpenMP gave us about 50 GB/s with all cores enabled and 80 GB/s with part of them disabled.