coremltools

I'm trying to figure out what the secret sauce for this is. It depends on https://github.com/apple/coremltools - is that the key trick or are there other important techniques going on here?

I’m not saying the M3 has a fundamental advantage. I’m saying that it’s unlikely to be as high a difference as is being stated in real world use. I don’t think a SOC at half the power budget is going to be magically more powerful. Regarding INT8, the frontend for the neural engine is CoreMLtools and it’s supported INT8 for a while , though their page does say the M4 has new int8-int8 acceleration https://apple.github.io/coremltools/docs-guides/source/opt-o... And one of the contributors to the repo saying the Neural Engine supports Int8 https://github.com/apple/coremltools/issues/929#issuecomment...

I'm the author of Private LLM. Looks like it's just become possible[1] to run quantized LLM inference using the ANE with iOS 18. I think there are some major efficiency gains on the table now. [1]: https://github.com/apple/coremltools/pull/2232

Yes, I specifically meant autoregressive LLMs. BERT style encoder only models, ViTs and CNNs ran perfectly fine. Yesterday's coremltools update[1] changes that. [1]: https://github.com/apple/coremltools/pull/2232

Some likely candidates are the existing open source tools their ML team has released, most of which either use PyTorch or are usable via PyTorch: - CoreNet, a training toolchain developed by their ML teams: https://github.com/apple/corenet. It's built on PyTorch, but uses MLX - MLX, Apples internal array processing framework which is C/C++/Swift (https://pypi.org/project/mlx/). - CoreML tools, their python package for accessing the lower-level CoreML APIs (https://github.com/apple/coremltools)

Instead of trying to integrate the whole stack of, say, pytorch, Apple's primary approach has been converting models to work with Apple's stack. https://github.com/apple/coremltools Clearly no one is going to be doing training or even fine tuning on Apple hardware at any scale (it competes at the low end, but at scale you invariably will be using nvidia hardware), but once you have a decent model it's a robust way of using it on Apple devices.

It is not. See https://developer.apple.com/documentation/coreml and https://github.com/apple/coremltools. The neural engine on the Apple A11 wasn't exposed to apps at least at launch, but that's no longer a thing on A12 onwards.

There are some areas where Apple is prioritizing getting developers on board with their hardware, and the neural engine seems like one of them. Over the past couple of years, coremltools [1], which is used to convert models from Tensorflow and other frameworks to run on Apple hardware (including the neural engine when available), has gone from a total joke to being quite good. I had to get a Keras model running on iOS a few months ago, and I was expecting to spend days tracking down obscure errors and writing lots of custom code to get the conversion to work -- but instead it was literally 3 lines of code, and it worked on the first try. [1] https://github.com/apple/coremltools

Some people are wondering if Tensorflow Lite would support CoreML / iPhone's neural engine. Tensorflow is not supported with coremltools (but I suppose Google could contribute a patch for this). I'm not sure about the CoreML libraries on the phone but from my understanding it may work. Keras is supported out of the box with the Tensorflow backend though. See https://medium.com/@JMangia/super-simple-end-to-end-test-of-... and https://github.com/apple/coremltools