are-we-fast-yet

A few years ago I made similar performance measurements based on the Are-we-fast-yet benchmark suite; here are some results: https://github.com/rochus-keller/Oberon/blob/master/testcase... There are always benchmarks where the one or the other might be faster; this is the reason why we use benchmark suites and geometric means of the factors to make the comparison more representative. In the above results, Node.js was twice as fast as C++ in the Storage benchmark, and equally fast in Richards and CD, but overall C++ was twice as fast. Implementations can differ significantly, which you confirmed with the two C++ implementations you measured. The Are-we-fast-yet suite includes guidelines (see https://github.com/smarr/are-we-fast-yet/blob/master/docs/gu...) for this purpose which must be followed by new implementations, and which assure comp

There are some results in the repository, e.g. one I published recently: https://github.com/rochus-keller/Are-we-fast-yet/blob/main/L... Or here: https://github.com/rochus-keller/Oberon/blob/master/testcase... The main repository only recently added a C++ implementation, but it was significantly slower than mine when I check last time (see https://github.com/smarr/are-we-fast-yet/issues/80). I mostly use the benchmarks to check how my compilers do compared to the big ones, or how the technologies I'm interested in evolve.

Have a look at the results and the code; there are benchmarks in the suite where the (ideomatic) C/C++ implementation is "only" twice as fast as the corresponding (idiomatic) Lua implementation, but on average (geomean of all factors) it's about five times as fast. The guidelines of the benchmark are pretty strict to enable fair comparisons (see https://github.com/smarr/are-we-fast-yet/blob/master/docs/gu...).

> Sure there's a small overhead to smart pointers Not so small, and it has the potential to significantly speed down an application when not used wisely. Here are e.g. some measurements where the programmer used C++11 and did everything with smart pointers: https://github.com/smarr/are-we-fast-yet/issues/80#issuecomm.... There was a speed down between factor 2 and 10 compared with the bare-bone C++98 implementation. Also remember that smart pointers create memory leaks when used with circular references, and there is an additional memory allocation involved with each smart pointer. > Garbage collection has an overhead too of course The Boehm GC is surprisingly efficient. See e.g. these measurements: https://github.com/rochus-keller/Oberon/blob/master/testcase.... The same benchmark suite as above is compared with different versions of Mon

> you are suggesting that it'd be the other way around, and I don't see why that'd be the case. That's just what I observed, and I concluded from the observations that Smalltalk - like Python - must be more difficult to JIT compile than Self or JS for some reason. I asked Stefan Marr for his most recent measurement results, and he was so friendly to post some reports: https://github.com/smarr/are-we-fast-yet/issues/85. As it seems the Graal/Truffle based implementation of SOM (a Smalltalk dialect) is now as fast as Node.js (V8), at least the AST version on the Graal enterprise edition; the Graal community editions of the SOM VMs are still slower though (about the same performance as Cog/OpenSmalltalk by estimate).

> So some languages/VMs apparently are more amenable than others. I thought Self was supposed to be harder to optimise due to using prototypes, accessor methods rather than distinguishing instance variable access, and generally fewer other tricks. Frankly I don't know what the problem is here, but if it was a language issue, then Self should be slower than Smalltalk - you are suggesting that it'd be the other way around, and I don't see why that'd be the case. Mario Wolczko implemented Smalltalk in Self, yielding a faster Smalltalk than the commercial offerings <http://www.merlintec.com/download/mario.pdf>. The are-we-fast-yet repo has benchmarks for SOMns (SOM modified to run Newspeak, using Truffle for compilation) which outperforms Node <https://github.com/smarr/are-we-fast-yet/blob/master/docs/pe...>. > Otherwise it's very difficult b

That's a very interesting article, thanks. Interesting to note that Cython is only about twice as fast as Python 3.10 and only about 40% faster than Python 3.11. The official Python site advertises a speedup of 25% from 3.10 to 3.11; in the article a speedup of 60% was measured. It therefore usually makes sense to measure different algorithms. Unfortunately there is no Python or C++ implementation yet for https://github.com/smarr/are-we-fast-yet.

> Software benchmarks are super subjective. No, they are not, but they are just a measurement tool, not a source of absolute truth. When I studied engineering at ETH we learned "Who measures measures rubbish!" ("Wer misst misst Mist!" in German). Every measurement has errors and being aware of these errors and coping with it is part of the engineering profession. The problem with programming language benchmarks is often that the goal is to win by all means; to compare as fairly and objectively as possible instead, there must be a set of suitable rules adhered to by all benchmark implementations. Such a set of rules is e.g. given for the Are-we-fast-yet suite (https://github.com/smarr/are-we-fast-yet).

I am aware of the various published test results where CoreCLR shows fantastic speed-ups compared to Mono, e.g. when calculating MD5 or SHA hash sums. But my measurements based on the Are-we-fast-yet benchmark suite (see https://github.com/smarr/are-we-fast-yet and https://github.com/rochus-keller/Oberon/tree/master/testcases/Are-we-fast-yet) show a completely different picture. Here the difference between Mono and CoreCLR (both versions 3 and 5) is within +/- 10%, so nothing earth shattering. Here are my measurement results: https://github.com/rochus-keller/Oberon/blob/master/testcases/Are-we-fast-yet/Are-… comparing the same benchmark on the same machine run under LuaJIT, Mono, Node.js and Crystal. https://github.com/rochus-keller/Oberon/blob/master/testcases/Are-we-fast-yet/Are-…

> Is it something to do with the extreme dynamic nature of the language? Smalltalk/SOM, JavaScript and Lua share the "extreme dynamic nature" but still have much faster JIT implementations. From my reports you can see that V8 (Node) is about ten times and LuaJIT about five times faster than MRI Ruby 3 JIT. Here are some other performance results using the same benchmark suite: https://github.com/smarr/are-we-fast-yet/blob/master/docs/pe.... Also TruffleRuby is reported to perform much better than MRI Ruby JIT (see e.g. https://pragtob.wordpress.com/2020/08/24/the-great-rubykon-b...).

I did some measurements recently with 3.0.0 RC1 compared to 2.1.0 based on https://github.com/smarr/are-we-fast-yet. Here are the results: http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub.... Ruby 3.0 without JIT is factor 1.2 faster than Ruby 2.1.0; Ruby 3.0 with JIT is only factor 1.4 faster than Ruby 2.1.0. Crystal is about factor 20 faster.

In case someone is interested in the (JIT) performance of 3.0 RC 1 compared to version 2.7.2, here are some measurement results based on the are-we-fast-yet benchmark suite: http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub... and http://software.rochus-keller.ch/are-we-fast-yet_crystal_rub.... I see only a slight speedup of the 3.0 (JIT) compared to 2.7 (less than 15%). See here for more information about the benchmark suite: https://stefan-marr.de/papers/dls-marr-et-al-cross-language-... and https://github.com/smarr/are-we-fast-yet. In contrast to other benchmarks the focus is on ideomatic language use and representative results (i.e. not just some random micro benchmarks) suited for inter-language comparisons.

There are already a couple of cross-language comparisons between Crystal and other languages based on (arbitrary) micro benchmarks. The Are-we-fast-yet benchmark suite (see https://github.com/smarr/are-we-fast-yet) includes representative benchmark implementations of different languages, whereby the authors consciously paid attention to an idiomatic use of the respective language. See here for more information about the implementation: https://stefan-marr.de/papers/dls-marr-et-al-cross-language-compiler-benchmarking-…. I run the benchmarks for some of the supported languages on my i386 Linux machine. Here are the results: http://software.rochus-keller.ch/are-we-fast-yet_crystal_ruby_lua_node_i386_result…. The results are normalized to Crystal. The geometric mean of the normalized values shows that Crystal is

https://github.com/smarr/are-we-fast-yet It is not about optimal performance. Ruby, or specifically MRI here, is anywhere between 20-100x slower. And there are other dynamic languages that is much faster then Ruby. If Ruby is only 2-10x slower no one would complain. But right now it is so far off the chart that is why you see a shrinking usage of Ruby.