DMdmw_ng·Oct 10, 2020
The parser from the article's performance is closer to 360mb/sec, the 2gb/sec is only achieved by over-allocating an incredible amount of hardware. CSV parsing is such a simple problem there is no reason to waste a whole machine on it, and in many contemporary scenarios (AWS Lambda), it's simply not possible to waste a whole machine on it.
I never finished packaging and writing tests, but https://github.com/dw/csvmonkey hits 1.9gb/sec on certain data sets with one thread, and that implementation is not even representative of the limits of single thread performance. https://github.com/geofflangdale/simdcsv follows a similar approach to simdjson, where multiple passes over the input are used, but with lower latency vector instructions (csvmonkey used one of the slowest). Character set decoding is also fused into the parser. There are no ben
Read the rest
chmarks published in that repo, but I expect the approach trounces my effort even if the current implementation doesn't.
Why be satisfied saturating one NVMe drive when a few threads could saturate the PCIe bus instead?