neural-engine

Mobile TPUs/NPUs: Pixel 6+ phones have TPUs (in addition to CPUs and an iGPU/dGPU). Tensor Processing Unit > Products > Google Tensor https://en.wikipedia.org/wiki/Tensor_Processing_Unit TensorFlow lite; tflite: https://www.tensorflow.org/lite From https://github.com/hollance/neural-engine : > The Apple Neural Engine (or ANE) is a type of NPU, which stands for Neural Processing Unit. From https://github.com/basicmi/AI-Chip : > A list of ICs and IPs for AI, Machine Learning and Deep Learning

If Apple would wake up to what's happening with llama.cpp etc then I don't see such a market in paying for remote access to big models via API, though it's currently the only game in town. Currently a Macbook has a Neural Engine that is sitting idle 99% of the time and only suitable for running limited models (poorly documented, opaque rules about what ops can be accelerated, a black box compiler [1] and an apparent 3GB model size limit [2]) OTOH you can buy a Macbook with 64GB 'unified' memory and a Neural Engine today If you squint a bit and look into the near future it's not so hard to imagine a future Mx chip with a more capable Neural Engine and yet more RAM, and able to run the largest GPT3 class models locally. (Ideally with better developer tools so other compilers can target the NE) And then imagine it does that while leaving the

My question too. This semi-answer on the page seems to contradict itself (source: https://github.com/hollance/neural-engine/blob/master/docs/p... ): "> Can I program the ANE directly? Unfortunately not. You can only use the Neural Engine through Core ML at the moment. There currently is no public framework for programming the ANE. There are several private, undocumented frameworks but obviously we cannot use them as Apple rejects apps that use private frameworks. (Perhaps in the future Apple will provide a public version of AppleNeuralEngine.framework.)" The last part links to this bunch of headers: https://github.com/nst/iOS-Runtime-Headers/tree/master/Priva... So might it be more accurate to say you can program it directly, but won't end up with something that can be distributed on the app store?