hyperfine

ecially if your cores are have very different maximum frequency) * Warns if thermal throttling is detected during the run * Warns if an inappropriate CPU governor is enabled * Locks the program into RAM (probably hard to do without some sort of help from the program) * Enables realtime priority if available (e.g., if isolcpus= is not enabled, or you're not on Linux) Of course, sometimes you would _want_ to benchmark some of these effects, and that's fine. But most people probably won't, and won't know that they exist. I may easily have forgotten some. On the flip side (making things more random as opposed to less), something that randomizes the initial stack pointer would be nice, as I've sometimes seen this go really, really wrong (renaming a binary from foo to foo_new made it run >1% slower!).

g wrong with computing average/min/max, this is not all hyperfine does. We also compute modified Z-scores to detect outliers. We use that to issue warnings, if we think the mean value is influenced by them. We also warn if the first run of a command took significantly longer than the rest of the runs and suggest counter-measures. Depending on the benchmark I do, I tend to look at either the `min` or the `mean`. If I need something more fine-grained, I export the results and use the scripts referenced above. > At that rate, one could just as well use a shell script and eyeball the results. Statistical analysis (which you can consider to be basic) is just one reason why I wrote hyperfine. The other reason is that I wanted to make benchmarking easy to use. I use warmup runs, preparation commands and parametrized benchmarks all the time. I also frequently use the Markdown export or the JSON export to generate graphs or histograms. This is my personal experience. If you are not interested in all of these features, you can obviously "just as well use a shell script".

/ discovery of those options.

benchmarks are run with hyperfine, a CLI benchmarking tool: https://github.com/sharkdp/hyperfine astral-sh/rye was mitsuhiko/rye, and rye wraps uv with "fallback to unearth [resolvelib] and pip-tools": https://github.com/astral-sh/rye https://lucumr.pocoo.org/2024/2/15/rye-grows-with-uv/ So, TIL it looks like rye for python and pixi for conda (python, r, rust, go,) are the latest tools for python packaging, not apt packaging. fpm is one way to create deb packages from python virtualenvs for install with the new APT solver, which resulted in this research.

tem: 10.0 ms] Range (min … max): 12.1 ms … 24.8 ms 241 runs Benchmark 5: bash -c '' Time (mean ± σ): 4.9 ms ± 0.8 ms [User: 2.8 ms, System: 1.8 ms] Range (min … max): 2.6 ms … 6.4 ms 552 runs Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options. Benchmark 6: echo Time (mean ± σ): 1.2 ms ± 0.3 ms [User: 0.7 ms, System: 0.3 ms] Range (min … max): 0.8 ms … 2.1 ms 2237 runs Summary echo ran 2.41 ± 0.89 times faster than perl -e '' 4.17 ± 1.39 times faster than bash -c '' 16.37 ± 5.91 times faster than php -r '' 20.74 ± 7.64 times faster than python -c pass 74.21 ± 22.49 times faster than node -e '' (tested with node 21.5.0, python 3.11.6, perl 5.38.1, php 8.2.14, and bash 5.2.21 on Ryzen 2700X) [1]: https://github.com/sharkdp/hyperfine

I encountered the following error: bash: /root/.inshellisense/key-bindings.bash: line 1: syntax error near unexpected token `$'{\r'' 'ash: /root/.inshellisense/key-bindings.bash: line 1: `__inshellisense__() { Due to this issue, I am unable to perform a performance test using hyperfine. The version of Bash available in this Docker image is 5.2.15(1)-release. I verified that the content of /root/.inshellisense/key-bindings.bash is exactly the same as https://github.com/microsoft/inshellisense/blob/main/shell/k...