neural-engine

At the release of Apple Silicon, there was this repos https://github.com/hollance/neural-engine That reference lot's of discovery and reverse engineer on the ANE.

MPS is okish, is this is what you mean. ANE and CoreML is kind of meh for most use cases, great for some very specific tasks. https://github.com/hollance/neural-engine/blob/master/docs/a...

It’s unfortunate that this announcement is still unspecific about what they improved in the Neural Engine. Since all we know about the Neural Engine comes from Apple papers or reverse engineering efforts (https://github.com/hollance/neural-engine), it’s plausible that they addressed some quirks to enable better transformer performance. They have written quite interesting papers on transformers on the Neural Engine: - https://machinelearning.apple.com/research/neural-engine-tra... - https://machinelearning.apple.com/research/vision-transforme... Things have definitely gotten better with MLX on the software side, though it still seems they could do more in that area (let’s see what the M5 Max brings). But even if they made big strides here, it won’t help previous generations, and the main thing limiting Apple Intelligence (in my opinion) wi

You can help with the reverse engineering of Apple Silicon done by a dozen people worldwide, that is how we find out the GPU and NPU instructions[1-4]. There is over 43 trillion float operations per second to unlock at 8 terabit per second 'unified' memory bandwidth and 270 gigabits per second networking (less on the smaller chips).... [1] https://github.com/AsahiLinux/gpu [2] https://github.com/dougallj/applegpu [3] https://github.com/antgroup-skyward/ANETools/tree/main/ANEDi... [4] https://github.com/hollance/neural-engine You can use a high level APIs like MLX, Metal or CoreML to compute other things on the GPU and NPU. Shadama [5] is an example programming language that translates (with Ometa) matrix calculations into WebGPU or WebGL APIs (I forget which). You can do exactly the same with the MLX, Metal or CoreML APIs and only pay ar

> Then we'll see somewhat simpler, less general, and cheaper devices that do "AI" operations with as little excess hardware baggage as possible. Nice. Apple has already been doing this for a few years now. The NPU is totally different from the GPU or CPU on the die itself[1]. Nvidia is likely working on this as well, but I think a device that's a gaming/entertainment/crypto/AI bundle (i.e. sticking with the video card) is probably a better business move. [1] https://github.com/hollance/neural-engine/blob/master/docs/a...

~38 TOPS at fp16 is amazing, if the quoted number if fp16 (ANE is fp16 according to this [1] but that honestly seems like a bad choice when people are going smaller and smaller even at the higher level datacenter cards so not sure why apple would use it instead of fp8 natively) [1]: https://github.com/hollance/neural-engine/blob/master/docs/1...

yep. they have a neural engine that is separate from the CPU and GPU that does really fast matmuls https://github.com/hollance/neural-engine. it's basically completely undocumented and for programs like blas it has more horsepower than all the cores combined. meteorlake and zen4 (8000) also both have npus which are similar. https://www.pcmag.com/news/the-meteor-lake-npu-meet-intels-d...

Mobile TPUs/NPUs: Pixel 6+ phones have TPUs (in addition to CPUs and an iGPU/dGPU). Tensor Processing Unit > Products > Google Tensor https://en.wikipedia.org/wiki/Tensor_Processing_Unit TensorFlow lite; tflite: https://www.tensorflow.org/lite From https://github.com/hollance/neural-engine : > The Apple Neural Engine (or ANE) is a type of NPU, which stands for Neural Processing Unit. From https://github.com/basicmi/AI-Chip : > A list of ICs and IPs for AI, Machine Learning and Deep Learning

If Apple would wake up to what's happening with llama.cpp etc then I don't see such a market in paying for remote access to big models via API, though it's currently the only game in town. Currently a Macbook has a Neural Engine that is sitting idle 99% of the time and only suitable for running limited models (poorly documented, opaque rules about what ops can be accelerated, a black box compiler [1] and an apparent 3GB model size limit [2]) OTOH you can buy a Macbook with 64GB 'unified' memory and a Neural Engine today If you squint a bit and look into the near future it's not so hard to imagine a future Mx chip with a more capable Neural Engine and yet more RAM, and able to run the largest GPT3 class models locally. (Ideally with better developer tools so other compilers can target the NE) And then imagine it does that while leaving the

My question too. This semi-answer on the page seems to contradict itself (source: https://github.com/hollance/neural-engine/blob/master/docs/p... ): "> Can I program the ANE directly? Unfortunately not. You can only use the Neural Engine through Core ML at the moment. There currently is no public framework for programming the ANE. There are several private, undocumented frameworks but obviously we cannot use them as Apple rejects apps that use private frameworks. (Perhaps in the future Apple will provide a public version of AppleNeuralEngine.framework.)" The last part links to this bunch of headers: https://github.com/nst/iOS-Runtime-Headers/tree/master/Priva... So might it be more accurate to say you can program it directly, but won't end up with something that can be distributed on the app store?

This is an insidious argument because there's a grain of truth in it. Photos/videos are heavily post-processed by Apples 'Neural engine' (https://github.com/hollance/neural-engine) running on the Bionic chips, but this is done at the time of recording. Zooming the video to play it to the jury would just be simple linear scaling, and could be done on any device capable of playing & scaling a video. For example, the zoom or crop functions in VLC. > have artificial intelligence in them that allow things to be viewed through three-dimensions and logarithms a dangerous mix of incoherent buzzwords that's nearly correct.

The Apple Neural Engine (ANE) consists of a "16-core architecture capable of 11 trillion operations per second, the Neural Engine in M1 enables up to 15x faster machine learning performance. In fact, the entire M1 chip is designed to excel at machine learning, with ML accelerators in the CPU and a powerful GPU, so tasks like video analysis, voice recognition and image processing will have a level of performance never seen before on the Mac." - https://www.apple.com/uk/newsroom/2020/11/apple-unleashes-m1/ The ANE is powerful where it comes to dealing with matrices, which are also used within games dealing with physics like collision detection or even real-time ray-tracing. - https://github.com/hollance/neural-engine The unified memory also makes it easier for the ANE and GPU to work together, instead of the GPU graphics pipeline or CPU cr

It’s accessible through the CoreML framework. I know nothing about this and very little about ML in general, but I found an interesting GitHub page written by a consultant who seems to know a fair amount about it. https://github.com/hollance/neural-engine