esp-nn

Hopefully it will be just added to https://github.com/espressif/esp-nn on software side.

There are a range of ML acceleration possible on existing chips. The basic 4-wide 8 bit integer SIMD extensions in NEON is available on basically all ARM Cortex M4F chips, which is already available 8+ years. It gives 4-5x speedup for neural networks. The more recent ESP32-S3 has operations with up to 10x speedup, see https://github.com/espressif/esp-nn Then there are RISCV chips with neural network co processors like Kendryte K210. ARM has also defined a new set of extensions for NN acceleration, and reference designs for cores being ARM Cortex M85. Chips are becoming available this year. ST has announced they will have accelerators in several lines. There are dozens of startups creating accelerator designs and trying to pair them with MCUs. So we have a bit already, with much more to come in the years to come.

Great post with very interesting detail, thanks ! Another optimization could be to quantize the model, this transform all compute as int compute and not as floating point compute. You can lose some accuracy, but for any bigger model it's a requirement ! Espressif do a great job on the TinyML part, they have different library for different level of abstraction. You can check https://github.com/espressif/esp-nn that implement all low level layers. It's really optimized and if you use the esp32-s3 it will unlock a lot of performance by using the vector instructions.

The ESP32-S3 introduces vector instructions which are supported by TFLite already. See here - https://github.com/espressif/esp-nn#performance S3 also supports Octal PSRAM vs the single lane SPI PSRAM found on the older ESP32-CAM style boards, should be 8x the bandwidth with all else equal. So far I've only seen the octal PSRAM's available on WROOM modules and it looks likes there only support for Espressif branded Octal PSRAM chips at this point.