hyperfine

xactly the things that I want to "average out" by running the benchmark multiple times. > What's worse is when these tools start to remove "outliers". Hyperfine never removes outliers. What we do is to try and detect outliers. We do this by computing robust statistical estimates that specifically DO NOT assume a normal distribution (see https://github.com/sharkdp/hyperfine/blob/master/src/hyperfi... for details). We perform this outlier detection to warn users about potentially interfering processes or caching effects. Take a look at these results, for example: https://i.imgur.com/XRvE6Ys.png I benchmarked a file-searching program. The underlying distribution, while probably not normal, seems to be "well behaved" and I think that the sample mean and the sample standard deviation could be quantities with a reasonably predictive power. What you do NOT see in the histogram is a single outlier at 1.15 seconds, far outside the plot to the right. This was the first benchmark run where the disk caches were still cold. In such a case, hyperfine warns the user: Warning: The first benchmarking run for this command was significantly slower than the rest (1.152 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run. In conclusion, I am not quite sure how your critisism applies to hyperfine, but I'd be happy to get further feedback.