csvmonkey

The parser from the article's performance is closer to 360mb/sec, the 2gb/sec is only achieved by over-allocating an incredible amount of hardware. CSV parsing is such a simple problem there is no reason to waste a whole machine on it, and in many contemporary scenarios (AWS Lambda), it's simply not possible to waste a whole machine on it. I never finished packaging and writing tests, but https://github.com/dw/csvmonkey hits 1.9gb/sec on certain data sets with one thread, and that implementation is not even representative of the limits of single thread performance. https://github.com/geofflangdale/simdcsv follows a similar approach to simdjson, where multiple passes over the input are used, but with lower latency vector instructions (csvmonkey used one of the slowest). Character set decoding is also fused into the parser. There are no ben

I can't speak for this project, but my own for CSV files ( https://github.com/dw/csvmonkey ) provides a high level interface that allows the tokenized data to be manipulated in-place without full decoding. The interface exported in Python is that of a plain old dictionary with one added magical semantic (lazy decode on element access). The internal representation of the parse result is a simple fixed array of (ptr, size) pairs Methods like this are used for batch search / summation where only a fraction of the parsed data is actually relevant during any particular run. You'll find similar approaches used in e.g. the row format parser of a database like MongoDB or Postgres

As much as I love SQLite, and while it is open source, it is a single implementation that AFAIK has no published open specification. The only way to read an SQLite file is using SQLite, and in that respect, it is for many users just as closed as wrapping something in a word document. CSV isn't perfect, but it provides a ton of flexibility, for example, CSVs can be streamed or support parallel segmented download across a network with useful work possible during the transfer. The format is so simple that it can approach almost free to parse (see e.g. my own https://github.com/dw/csvmonkey ). CSV is also distinguished in that regular home users with spreadsheet programs can usually do most things a developer can do with the same file. For me user empowerment trumps all other goals in software, including warts. Things like JSON, XML or SQLite

Check out csvmonkey for a really fast C++ csv parser: https://github.com/dw/csvmonkey

While Pandas csv parser is quite slow anyway, the reason Pandas is particularly slow in this case is because it insists on applying type detection to every field of every row read. I have no clue how to disable it, but it's default behaviour. Parsing isn't actually a tough problem – https://github.com/dw/csvmonkey is a project of mine, it manages almost 2GB/sec throughput _per thread_ on a decade old Xeon

I'm not "making a claim" or inventing a heuristic, you can test this trivially: $ git clone https://....:x-oauth-basic@github.com/dw/csvmonkey.git Cloning into 'csvmonkey'... remote: Counting objects: 340, done. remote: Compressing objects: 100% (27/27), done. remote: Total 340 (delta 19), reused 27 (delta 10), pack-reused 303 Receiving objects: 100% (340/340), 138.93 KiB | 0 bytes/s, done. Resolving deltas: 100% (212/212), done. $ cat csvmonkey/.git/logs/HEAD 0000000000000000000000000000000000000000 c9d566bf167dcf3556008df58be37c4a27ff5062 David Wilson <dw@botanicus.net> 1497289486 +0100 clone: from https://....:x-oauth-basic@github.com/dw/csvmonkey.git If you perform a Git checkout on a web server e.g. as part of an Ansible script, and you embedded secrets in the repo URL (common enough, believe me)