CSV.jl

I posted this comment on lobste.rs when this came up:[1] It kind of looks like Julia’s CSV parser is cheating: https://github.com/JuliaData/CSV.jl/blob/9f6ef108d195f85daa5... It’s doing parallel parsing, but I’m pretty sure their technique won’t work for all inputs. Namely, they try to hop around the CSV data and chunk it up, and then parse each chunk in a separate thread AIUI. But you can’t do this in general because of quoting. If you read the code around where I linked, you can see they try to be a bit speculative and avoid common failures (“now we read the next 5 rows and see if we get the right # of columns”), but that isn’t going to be universally correct. It might be a fair trade off to make, since CSV data that fails there is probably quite rare. But either I’m misunderstanding their optimization or they aren’t being transparent a

EDIT: Be sure to read the reply from the author of the package. I do think Julia has promise, but it is still early days. In the last 2 months there have been problems with: * Throwing rows away: https://github.com/JuliaData/CSV.jl/issues/720 * Incorrect float parsing: https://github.com/JuliaData/CSV.jl/issues/714 At there are unfixed issues to do with parsing large files ( https://github.com/JuliaData/CSV.jl/issues/749 , https://github.com/JuliaData/CSV.jl/issues/734 ). While Julia has become much more stable than it used to be, it is still in many places an evolving system and you (in my limited experience) should be prepared to hit issues.