simdcsv

There are already a quite a few [0][1] CSV parsers that use SIMD, some in Rust, with a variety of approaches. I found simd-csv[1] to have a very interesting approach that leverages memchr to essentially "seek" for the next delimiter, reducing a lot of the overhead that a byte-by-byte CSV parser would have. However, as noted in the README, the creators of simd-csv explicitly chose not to use the classic pclmulqdq trick[2] that other libraries like simdjson use due to portability concerns. I set out to beat to simd-csv's implementation by building a parser more similar to Geoff Langdale's, using the pclmulqdq trick as well as optimized intrinsic usage for aarch64 platforms[3]. If anyone has feedback on the Rust code, or my usage of intrinsics, I would greatly appreciate it. [0] https://github.com/geofflangdale/simdcsv [1] https://github.com

Lemire himself also worked on a C++ CSV parser with Geoff Langdale that uses SIMD to accelerate the parsing. [1]. Also the fastest library in TFAs benchmark is Sep [2], which makes use of SIMD. [1] https://github.com/geofflangdale/simdcsv [2] https://github.com/nietras/Sep

shameless insertion: "raw csv processing at ~20GB/s" has been demonstrated in one my project as the tooling byproduct[1] based on Rayon(great Rust data parallelism library)[2] and wrapping of modified simdcsv[3] into one simple .rs. just a little more: 1. simdcsv has severe bugs, so do not use it beyond demo. 2. the processing model of simdcsv still has rooms to good improvements(estimated 2x more, a.k.a. ~40GB+/s in memory in single modern socket should be achievable in some scenarios(no heavy string to complex language object conversions)). [1] https://tensorbase.io/2020/08/04/hello-base.html#benchmark [2] https://github.com/rayon-rs [3] https://github.com/geofflangdale/simdcsv

The parser from the article's performance is closer to 360mb/sec, the 2gb/sec is only achieved by over-allocating an incredible amount of hardware. CSV parsing is such a simple problem there is no reason to waste a whole machine on it, and in many contemporary scenarios (AWS Lambda), it's simply not possible to waste a whole machine on it. I never finished packaging and writing tests, but https://github.com/dw/csvmonkey hits 1.9gb/sec on certain data sets with one thread, and that implementation is not even representative of the limits of single thread performance. https://github.com/geofflangdale/simdcsv follows a similar approach to simdjson, where multiple passes over the input are used, but with lower latency vector instructions (csvmonkey used one of the slowest). Character set decoding is also fused into the parser. There are no ben