blake2b-simd

I submitted this earlier. This github project (not mine) has pretty extensive benchmarks and the M1 makes a very strong showing. https://news.ycombinator.com/item?id=28346908 On another note, I was amazed to see the SHA-1 and SHA-256 (and the Blake2b) performance numbers when running the benchmark of minio’s blake2b-simd (over M1 Golang) on a 16GB M1 Air. For some reason, Go on M1’s SHA-1 and SHA-256 are getting orders of magnitude better numbers (~2.5GB/sec!) and even the Blake2b code is beating the numbers for the same code running on an intel Air (which uses the SIMD codepath). https://github.com/minio/blake2b-simd

There is AVX/AVX2 assembly for BLAKE2 at https://github.com/minio/blake2b-simd , benchmarked at a 1.8x-3.9x speedup.

Oh, interesting. I was inferring that from the Go code out there that uses machine code directly: https://github.com/minio/blake2b-simd/blob/master/compressAv... I guess it must be a recent addition.

Yes true. OpenSSL version gives a great boost. Rather than SHA512 we ended up using Blake2b - https://github.com/minio/blake2b-simd (also optimized with SIMD instructions) internally for bit-rot verification in https://github.com/minio/minio

Hello HN, Here is a bit more explanation about erasure code and bit-rot protection. Minio along with being S3 compatible also protects data against hardware failures and silent data corruption using erasure code and checksums. You may lose roughly half the number ((N/2)-1) of drives and still be able to recover the data. Minio uses Reed-Solomon code to stripe objects into N/2 data and N/2 parity blocks. This means that in a 12 drive setup, an object is striped across as 6 data and 6 parity blocks. You can lose as many as 5 drives (be it parity or data) and still reconstruct the data reliably from the remaining drives. Minio also supports bit-rot detection transparently which works along with erasure code using high speed BLAKE2 - https://github.com/minio/blake2b-simd hash based checksums.