laser

ch_matmul like what CuBLAS provides and is quite useful for convolution via matmul.

v time: 0.596 ms Min time: 21.769 ms Max time: 23.603 ms Perf: 624.037 GFLOP/s OpenBLAS benchmark Collected 10 samples in 0.216 seconds Average time: 21.340 ms Stddev time: 3.334 ms Min time: 19.346 ms Max time: 27.502 ms Perf: 663.359 GFLOP/s MKL-DNN JIT AVX512 benchmark Collected 10 samples in 0.201 seconds Average time: 19.775 ms Stddev time: 8.262 ms Min time: 15.625 ms Max time: 43.237 ms Perf: 715.855 GFLOP/s ``` Note: the Theoretical peak limit is hardcoded and used my previous machine i9-9980XE. It maybe that your BLAS library is not named libopenblas.so, you can change that here: https://github.com/mratsim/laser/blob/master/benchmarks/thir... Implementation is in this folder: https://github.com/mratsim/laser/tree/master/laser/primitive... in particular, tiling, cache and register optimization: https://github.com/mratsim/laser/blob/master/laser/primitive... AVX512 code generator: https://github.com/mratsim/laser/blob/master/laser/primitive... And generic Scalar/SSE/AVX/AVX2/AVX512 microkernel generator (this is Nim macros to generate code at compile-time): https://github.com/mratsim/laser/blob/master/laser/primitive... I'll come back later with details on how to use my custom HPC threadpool Weave instead of OpenMP (https://github.com/mratsim/weave/tree/master/benchmarks/matm...). As a side bonus it also has parallel nqueens implemented.

s C++ interop support C++17? C++20? Can you guarantee it will support C++27? Nim's implementation already does, on every single platform you'll be able to use C++27 on (as long as C++27 can compile modern C++ code; there had been backward incompatible changes along the C++ history). You can't just #include a C or C++ header and call it a day; You need to have a Nim compatible definition for any symbol (variable, macro, function, class, ...). There are tools that help you and make it almost as easy as #include, such as nimterop[0] and and nimline[1], and "c2nim" which is included with the Nim compiler is enough to generate the Nim definitions from the .h definitions (though it can't do crazy metaprogramming; if D can do that, then D essentially includes a C++ compiler. Which is a fine way to get perfect C++ compatibility - Nim does that) But Nim can also do the same for JS when treating JS as a backend. And it can basically do the same for Python, with nimpy[2] and nimporter, generating a single executable that works with your installed Python DLL (2.7, 3.5, 3.6, 3.7) - which is something not even Python itself can do. There was a similar Lua bridge, but I think that one is no longer maintained. > We also have Mir, which I haven't benchmarked for a while but was faster than OpenBLAS and Eugene There's quite a bit of scientific stack built natively with Nim. It is far from self-sufficient, but the ease with which you can use a C library makes up for it. I haven't used it, but Laser[3] is on par with or exceeds OpenBLAS speedwise, and generalizes to e.g. int32 and int64 matrix multiplication; Arraymancer[4] does not heve all of numpy's functionality but does have quite a few nice bits from scikit-learn, supports CUDA and OpenCL, and you can use numpy through nimpy if all else fails. Also notable is NimTorch[5]. laser and arraymancer are mostly developed by mratsim, who occasionally hangs out here on HN. D is a fine language, I used it a little in the D1 days, and it was indeed a "better C++" but did not deliver enough value to be worth it for me, so I stopped. I know D2 is much better, but I've already found my better C++ (and better Python at the same time!) in Nim, so I haven't looked at it seriously. [0] https://github.com/nimterop/nimterop [1] https://github.com/sinkingsugar/nimline [2] https://github.com/yglukhov/nimpy [3] https://github.com/numforge/laser [4] https://github.com/mratsim/Arraymancer [5] https://github.com/sinkingsugar/nimtorch

c allocation suitable for multithreading runtimes and embedded devices and able to display the actual graph at compiletime? - https://github.com/mratsim/Synthesis Can you describe x86 opcodes in a declarative way for a JIT assembler and do all the rex/modrm/sib computation at compile-time? - https://github.com/mratsim/photon-jit/blob/master/photon_jit... And the same thing for an emulator? - https://github.com/mratsim/glyph/blob/master/glyph/snes/opco... Can you implement async as a library? Can you emulate classes with ADTs to solve the expression problem, avoid cache misses and multithreading problem due to OOP and the double indirection due to the visitor pattern? - https://github.com/mratsim/trace-of-radiance/blob/master/tra... > Nim might have a speed overhead in certain tasks but that depends on a benchmark. Besides, does Nim have anything similar to NumPy which is actually faster? D does. It does: - https://github.com/mratsim/Arraymancer With CPU, OpenCL and Cuda backends (in differing state of maturity) and it's being used for data analysis for example ancestry prediction: - http://home.chpc.utah.edu/~u6000771/somalier-ancestry.html You can even run a neural network and see its training live: - https://github.com/Vindaar/NeuralNetworkLiveDemo AFAIK D doesn't have any plotting library while Nim has plot.ly integration and a ggplot2 port in pure Nim: - https://github.com/brentp/nim-plotly - https://github.com/Vindaar/ggplotnim And lastly nim can easily call Python: - https://github.com/yglukhov/nimpy or be called from Python: - https://github.com/Pebaz/nimporter

type-erasure to handle everything people can use a multithreading runtime for, same comparison on GEMM: https://github.com/mratsim/weave/tree/v0.4.0/benchmarks/matm... - More resources on the importance of memory bandwidth: optimization convolutions https://github.com/numforge/laser/wiki/Convolution-optimisat... - Optimizing matrix multiplication on GPUs: https://github.com/NervanaSystems/maxas/wiki/SGEMM, again it's all about memory and caches optimization - Let's switch to another domain with critical perf need, cryptography. Even when knowing the bounds of iterating on a bigint at compile-time, compiler are very bad at producing optimized code, see GCC vs Clang https://gcc.godbolt.org/z/2h768y - And crypto is the one thing where integer templates are very useful since you know the bounds. - Another domain? VM interpretation. The slowness there is due to function call overhead and/or switch dispatching and not properly using hardware prefetchers. Same thing, C++ constexpr doesn't help it's lower-level, see resources: https://github.com/status-im/nimbus/wiki/Interpreter-optimiz... Also all the polyhedral research, and deep learning compiler research including the Halide compiler, Taichi, Tiramisu, Legion, DaCE confirm that memory is the big bottleneck. Now since you want to stop on the theory and you mentioned HPC, pick your algorithm, it could be matrix multiplication, QR decomposition, Cholesky, ... Any fast C++ code (or C, or Fortran or Assembly) that you find will be fast because of careful memory layout and all level of caches, not constexpr. If you have your own library in one of those domains I would be also very happy to have a look. As a simple example, let's pick an out-of-place transposition kernel to transpose a matrix. Show me how you use constexpr and template metaprogramming to speed it up. Here is a detailed analysis on the impact of 1D-tiling and 2D tiling: https://github.com/numforge/laser/blob/master/benchmarks/tra..., throughput can be increased 4x with proper usage of memory caches.

I have had a difficult time since finding it two years ago ever coming up with an adequate reason for using it. Edit: I want to make one clarification. I think that Nim may actually have a bright future in ML, primarily due to the work of Mamy Ratsimbazafy. He has some libraries for HPC, Laser [0] and Arraymancer [1] that can smoke C by several orders of magnitude. Combined with Weave [2], the multi-threaded runtime he built for Nim, that opens the doors for ridiculously performant ML. I think the only the other language that would give Nim a run for its money here is Julia, I really dig the work that has been going on there for ML as well. [0] https://github.com/numforge/laser [1] https://github.com/mratsim/Arraymancer [2] https://github.com/mratsim/weave

low_level_design_considerations.md and challenges.md that should be interesting as well. Unfortunately I didn't go beyond the proof of concept as I need to revamp the Nim multithreading runtime to improve composition of parallel linear algebra kernels. One thing for sure is that OpenMP is today something that needs to be workaround due to it's lack of composability and GCC implementation is a contention point due to using a single queue protected by a lock. While traditional linear algebra algorithms can deal with static distribution and no load balancing, some algorithms would hugely benefits from a more flexible task system, for example Beam Search.

tion C[i, j] += A[i, k] * B[k, j] - We should have enough SIMD registers left to hold the corresponding sections of A and B (at least 4, 2xA and 2xB for FMAs) On x86-64 X SIMD registers that can issue 2xFMAs per cycle: - NbVecs is 2 minimum - RegsPerVec = 2 * NbVecs => 4 minimum (for A and for B) - NR = NbVecs * NbScalarsPerSIMD - C: MR*NR and uses MR*NbVecs SIMD registers - MR*NbVecs + RegsPerVec <= X -> MR*NbVecs + 2 * NbVecs <= X -> (MR+2) * NbVecs <= X Some solutions: - AVX with 16 registers: - MR = 6, NbVecs = 2 FP32: 8xFP32 per SIMD --> NR = 2x8 ukernel = 6x16 FP64, ukernel = 6x8 - MR = 2, NbVecs = 4 FP32: 8xFP32 per SIMD --> NR = 4x8 ukernel = 2x32 FP64, ukernel = 2x16 - AVX512 with 32 registers - MR = 6, NbVecs = 4 FP32 ukernel = 6x64 FP64 ukernel = 6x32 - MR = 2, NbVecs = 8 FP32 ukernel = 2x128 FP64 ukernel = 2x64 - MR = 14, NbVecs = 2 FP32 ukernel = 14x32 FP64 ukernel = 14x16 And in-depth overview of the lowest level details is available in the paper Automating the last mile for High Performance Dense Linear Algebra[5]. In short, the compiler is completely unable to deal with this, and a high performance computing compiler should give an escape hatch to allow hand optimization like Halide[6] or Tiramisu do[7]. [1]: https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1... [2]: https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1... [3]: https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1... [4]: https://github.com/numforge/laser/blob/e660eeeb723426e80a7b1... [5]: https://arxiv.org/pdf/1611.08035.pdf [6]: https://halide-lang.org/ [7]: http://tiramisu-compiler.org/