NNPACK

I'm playing a lot with VM, interpreters, JIT and recently compilers in Nim. Here is a JIT brainfuck VM: https://github.com/mratsim/jitterland Nim macros are extremely useful for assemblers and emulators as contrary to many codebases which needs to use an intermediate code generation step via text files [1][2][3] you can directly use them with type checking and all Nim facilities while keeping an easy to read syntax: https://github.com/numforge/laser/blob/e898f027/laser/photon... [1]: Xbyak - https://github.com/herumi/xbyak/blob/v5.751/gen/gen_avx512.c... [2]: NNPack + PeachPy - https://github.com/Maratyszcza/NNPACK/blob/af40ea7d/src/x86_... [3]: Apache Milagro Crypto - https://github.com/apache/incubator-milagro-crypto/blob/d43a...

Thanks for the pointer. The full softmax implementation is here [1]. I have not read the code, but I can trust the developer to have a very fast implementation. Nonetheless, I don't think the reference implementation in your original link is optimal. Exp is expensive and should not be called twice (EDIT: unless you can show a benchmark to prove me wrong). [1]: https://github.com/Maratyszcza/NNPACK/blob/master/src/x86_64...

Oops, here is the actually used vectorized expf (similar to the one I linked, but with optional unrolling): https://github.com/Maratyszcza/NNPACK/blob/master/src/x86_64...

This is the reference implementation of softmax, i.e. implementation used as a reference in unit tests. It is designed to be simple, readable, and correct, which is why I linked it here. Optimized implementation is in assembly (PeachPy). See https://github.com/Maratyszcza/NNPACK/blob/master/src/x86_64... for the vectorized expf

>>> Softmax can be implemented as a composition of primitive TensorFlow ops (exponent, reduction, elementwise division, etc.): softmax = exp(logits) / reduce_sum(exp(logits), dim) No, it can not be implemented this way, it is numerically unstable, and will produce NaNs if any input is greater than ~88.7. Luckily, it is also not how its implemented in Tensorflow: https://github.com/tensorflow/tensorflow/blob/2c8d0dca978a24... For a clean (and more efficient) C version of this algorithm, take a look at NNPACK reference implementation: https://github.com/Maratyszcza/NNPACK/blob/master/src/ref/so...

https://github.com/Maratyszcza/NNPACK/blob/master/src/x86_64... (from https://github.com/Maratyszcza/NNPACK) is also a pretty reasonable AVX2 GEMM implementation written in a Python assembler, it's a bit easier to follow than the OpenBLAS kernels and has very reasonable performance vs OpenBLAS/MKL. I second the guidance to just use a BLAS library. I'd guess you'd see a 2-10x speedup at least. It looks like you're only implementing a classical MLP library (which is fine for learning, but not the best choice for something like this, integrating VW into Redis would be a much better choice IMO).

im2col + sgemm on CPU as in Caffe for instance is really slow; you are heavily penalized for extra memory traffic and the sgemm tile sizes are probably not well tuned for the problem size at hand. At the roofline of performance, the difference in both mem b/w and arithmetic throughput between CPU and GPU is only 5-10x (for fp32, Pascal fp16 is a different story of course), and proper implementations on the CPU will get you there. https://github.com/Maratyszcza/NNPACK

I'm enthusiastic that Intel is working on optimizing Torch's Tensor and nn libraries, and optimizing Neural Nets in general. This repo is most definitely not ready for primetime. Two comments: 1. I haven't seen an Intel team that does non-trivial DNN optimizations, almost all the (multiple) teams that have worked on optimizing neural networks have worked on trivial stuff like adding OpenMP optimizations to existing code, and integrating MKL into existing frameworks. Usually adding OpenMP optimizations isn't as relevant, because one uses CPUs for neuralnet inference, on a server where multiple processes are running. Per-core efficiency is much more important than cute threadpool optimizations like OpenMP which look great in micro-benchmarks. The meaty and practically relevant optimizations have come from folks in the community like Andrew