nervanagpu

I wonder how much of this was motivated by Nervana System's successes building their own assembler for Neon [0], or if perhaps this work started before it. In either case more openness for low level hardware optimizations is always welcome and beneficial to the community. [0] https://github.com/NervanaSystems/nervanagpu

Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do the fastest. There has been quite some research on reducing the bit depth further the last couple of years, see e.g. http://arxiv.org/abs/1502.02551 and http://arxiv.org/abs/1412.7024. The current generation of GPUs already has limited support for half-precision operations, and the tools for using these operations in neural networks (dot product, convolution implementation) are starting to become available as well, which is awesome (see e.g. https://github.com/NervanaSystems/nervanagpu). NVIDIA themselves have also noticed and better hardware support for low-preci