benchmark

> it always makes me wonder how developers these days reason about what the code is actually going to compile to and perform like What will it compile to? Check out Compiler explorer [0]. What will it perform like? Use benchmarks [1] [2]. Remember that benchmarks heavily depend on what system you're using. Benchmarks for generically-compiled software on modern hardware will look very different than benchmarks for the same software but hyper-optimized to run on 10-year-old hardware. So: if it's not tested, it's not Engineered. For tests, I strongly prefer GTest [3], for a fairly consistent API across benchmarking, testing, and mocking. > there's a constant influx of new syntax and new concepts in C++ Yes, but you don't have to use all of it. You can still use older C++ if you really want to. I wouldn't recommend it though. I think the "swe

It's also fairly normal to do this. Google benchmark [1] even warns you about this. [1] https://github.com/google/benchmark

This is important work, at least partially addressed in other benchmark harnesses (like github.com/google/benchmark has DoNotOptimize and ClobberMemory https://github.com/google/benchmark/blob/7fad964a94da9364af1...). By and large this whole area is criminally under-examined, IMO. What I'd really like is for someone to put together a suite of statistical tests that reject poor performance measurements. At a bare minimum your performance samples should look like they come from IID random variables when you take them. If not, you haven't reached steady state, or there are correlations between repeated runs, etc.

> k6 is a modern load testing tool, building on Load Impact's years of experience in the load and performance testing industry. It provides a clean, approachable scripting API, local and cloud execution, and flexible configuration. This sounds more like integration testing than unit testing. Something like Google's micro-benchmarking library [1] is more like unit testing. [1] https://github.com/google/benchmark

For each size shown on the x-axis, every hashing call using exactly that size? It would be interesting to also have a test where an average size was chosen, but where the actual sizes varied randomly around that average - because that's a common case in real world code, and the results can be very different since branch prediction won't be perfect any more as it is with the fixed size tests. Also, latency usually means feeding the output of the function back into the input of the next call. Throughput is what you get when you don't do that but you still need to consume the result, so it's not optimized away. You can use something like DoNotOptimize here: https://github.com/google/benchmark/blob/master/include/benc...

Because of noise in general, "best case" seems always like the best metric to me. Over a large number of run, you're likely to hit the "perfect" measurement with on a microbenchmark. Otherwise, for an "adaptive" number of runs till enough time is spent to have some "confidence" on the measure, I've been fairly happy with: https://github.com/google/benchmark/