HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
BurntSushi
rust-csv
Rust2014–2020
BurntSushi / rust-csv

rust-csv

A CSV parser for Rust, with Serde support.

Languages & CompilersData & DatabasesLibraries & SDKs
Explore on GitHubRead Hacker News discussions
7Distinct people
16Mentions
0Explicit recommenders
2KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A CSV parser for Rust, with Serde support.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:03:16 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year16 dated mentions

Observed mentions: 2014-03-26 19:37:53 UTC to 2020-05-15 11:43:41 UTC.

01232014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments20142015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments20172018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2020

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC312
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC303
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC202
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC303
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC202
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC211
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC101

What Hacker News says

16 sources
burntsushi·Sep 9, 2018
The LuaJIT idea is interesting, I've certainly been impressed by it in the past, and can agree it is to some extent something that dispels myths like "statically typed languages are always faster than unityped languages." But if you instead interpret that as a first approximation, then it's fairly accurate IMO. In the interest of cutting to the chase, I'll try to explain some of the high level ideas of why the CSV parser is fast, and typically faster than any other CSV parser I've come across. Firstly, it is implemented by a hand-rolled DFA that is built from an NFA. The NFA is typically what most robust CSV parsers use, and it is quite fast, but it suffers from the overhead of moving through epsilon transitions and handling case analysis that is part of the configuration of the parser (i.e., delimiter, quote, escaping rules, etc.). It se
Read the rest
ems to me like this concept could be carried over to LuaJIT. Secondly, the per-byte overhead of the DFA is very low, and even special cases[1] some transitions to get the overhead even lower. If you were doing this in pure Python or Lua or really any unityped language, I would be very skeptical that you could achieve this because of all the implicit boxing that tends to go on in those languages. Now, if you toss a JIT in the mix, I kind of throw my hands up. Maybe it will be good enough to cut through the boxing that would otherwise take place. From what I've heard about Mike Pall, it wouldn't surprise me! If the JIT fails at this, I'm not sure how I'd begin debugging it. I kind of imagine it's like trying to convince a compiler to optimize a segment of code in a certain way, but only harder. Thirdly, a critical aspect of keeping things fast that bubbles all the way up into the xsv application code itself is the amortization of allocation. Namely, when xsv iterates over a CSV file, it reuses the same memory allocation for each record[2]. If you've written performance sensitive code before, then this is amateur hour, but I personally have always struggled to get these kinds of optimizations in unityped languages because allocation is typically not a thing they optimize for. Can a JIT cut through this? I don't know. I'm out of my depth. But I can tell you one thing for sure: in languages like Rust, C or C++, amortizing allocation is a very common thing to do. It is straight-forward and never relies on the optimizer doing it for you. There are some different angles to take here though. For example, unityped languages tend to be garbage collected, and in that environment, allocations can be faster which might make amortization less effective. But I'm really waving my hands here. I'm just vaguely drawing on experience. Anyway, I think it's kind of counter productive to try to play the "knows better than the hivemind" role here. There are really good solid reasons why statically typed languages tend to out-perform unityped languages, and just because there is a counter example in some cases doesn't make those reasons any less important. I think I could also construct an argument around how statically typed languages make it easier to reason about performance, but I don't quite know how to phrase it. In particular, at the end of the day, both cases wind up relying on some magic black box (a compiler's optimizer or a JIT), but I'm finding it difficult to articulate why that isn't the full story. [1] - https://github.com/BurntSushi/rust-csv/blob/546291a0095a2537... [2] - https://github.com/BurntSushi/xsv/blob/9574d89634031259802dd...
In XSV: A fast CSV command-line toolkit written in Rust
burntsushi·Sep 9, 2018
I would definitely love to be proven wrong, because if I am, I am certain I would learn something new. I am pretty comfortable with Python, so I am pretty comfortable saying that you could not write a tool as fast as xsv in Python without writing some substantial portion of it in C. Using Python's standard library CSV parser is certainly fair game, but writing more C code on top of that which included the loop over records in a CSV file feels like a cheat. The problem here is that you've already lost because xsv's CSV parser is faster than Python's parser at the C level[1]. :-) Assuming I haven't made any silly gaffes in xsv itself, I don't see how you're going to make up the difference. Because of that, I am willing to extend a handicap: if you can get within the same order of magnitude as xsv while doing the same amount of work in a rob
Read the rest
ust way, then I think I might even learn something there too. :-) I am less certain about PyPy, LuaJIT or Node, but I am certain I'd learn something if you proved me wrong. Note that a problematic part of this challenge is that your program would need to correctly handle CSV in a robust manner. It is very easy to write a very fast CSV parser that doesn't correctly handle all of the corner cases. Python's CSV parser certainly passes that test, but I don't know if Node's or Lua's CSV parser does because I've never used them. [1] - https://github.com/BurntSushi/rust-csv/blob/master/csv-core/...
In XSV: A fast CSV command-line toolkit written in Rust
burntsushi·Jun 14, 2017
> IIRC UTF8 stuff has been a perf bottleneck in the url crate Right, and this isn't surprising at all. It's not at all obvious to me that it's premature. It's really easy for from_utf8 to show up in a profile, and I've had occasion to either work around it safely (perhaps by staying in `&[u8]` land) or by resorting to `unsafe`. I did the latter in the CSV crate for managing `StringRecord`s.[1] The record is stored contiguously in memory, which means that even if the entire record is valid UTF-8 as a whole, then each individual field may not be. Checking UTF-8 validity on each field or on access leads to a slow down. (There are benchmarks.) It's much nicer to code a fast path that checks if the entire record is all ASCII. (Because then you can assume each field is valid UTF-8.) [1] - https://github.com/BurntSushi/rust-csv/blob/34e957e63bb9
Read the rest
2853...
In Bugs You'll Probably Only Have in Rust
burntsushi·Jun 10, 2017
The `csv` crate uses allocation, but `csv-core` does not. I think it would help if you looked at the `read_field` API in `csv-core`: https://docs.rs/csv-core/0.1.2/csv_core/struct.Reader.html#m... The key is that the parser can start and stop at any point in time. So if you give it a fixed size stack allocation and it isn't big enough to hold the data, then `read_field` will return `OutputFull`.[1] At that point, it's up to the caller to figure out what to do. If caller literally cannot dynamically allocate memory at all, then the caller might choose to return an error. Otherwise, the caller might choose to allocate more space and then ask the CSV parser to pick back up where it left off. With that said, while the `read_field` API is relatively simple, it's actually a bit slower than one would hope because it has to return to the caller e
Read the rest
very time a single field is read. So there's some overhead there. For that reason, there is also a `read_record` API, which is more complicated to use but is faster and preserves the zero-allocation property. It's faster because it reads an entire record at once. The higher level `csv` library uses the read_record API[3], and it will expand its allocation when necessary. The allocation bit isn't terribly important for performance, so long as you can amortize it (which is covered my blog post towards the end). The real trick to making it fast is the DFA. The DFA provides a way to do a very small constant amount of work per byte, which means that the configurability of the CSV parser has zero overhead because it's baked into the DFA. Compare that with the traditional NFA parser[4], and you can see how each state might need to step through multiple conditional checks. [1] - https://docs.rs/csv-core/0.1.2/csv_core/enum.ReadFieldResult... [2] - https://docs.rs/csv-core/0.1.2/csv_core/struct.Reader.html#m... [3] - https://github.com/BurntSushi/rust-csv/blob/34e957e63bb92853... [4] - https://github.com/BurntSushi/rust-csv/blob/master/csv-core/...
In Rust and CSV parsing
burntsushi·Jun 10, 2017
It's buried in the docs of csv-core[1], mostly because it's rarely useful in my experience. But it's a good question, so it probably should be featured more prominently. The implementation follows in the footsteps of Python's CSV parser, where it prefers a parse over returning an error. I'd like to add a strict mode[2], but I haven't perceived much demand for it. For writing and RFC 4180 compliance, see: https://docs.rs/csv-core/0.1.2/csv_core/struct.Writer.html#r... (The higher level `csv::Writer` will, by default, return an error when writing flexible records. Of course, that can be disabled.) [1] - https://docs.rs/csv-core/0.1.2/csv_core/struct.Reader.html#r... [2] - https://github.com/BurntSushi/rust-csv/issues/77
In Rust and CSV parsing
peterhj·Oct 19, 2016
I'm not sure that there's a mature Rust alternative to "all of the above" quite yet. Some things that I've had to build for my own work (not always "built," sometimes I just wrote FFI bindings) include: * N-D dense arrays, fashionably simply called tensors these days ;) (this I wrote from scratch to suit my own needs, but there are several implementations of this in Rust, as well as in C++) * linear algebra (bindings to BLAS, LAPACK etc.) * numerical integration (bindings to GSL or QUADPACK) * optimization and machine learning (this I work on day-to-day) * neural networks (I also work on this) * multicore/manycore/GPU/distributed support (and this) There's also the more practical aspects of using Pandas/iPython (Jupyter)/Spark for data science, including ETL, interactive work, and plotting, which IMO are not quite there yet in the Rust ec
Read the rest
osystem. But I think that a lot of the pieces are there, e.g. bleeding fast CSV parsing libraries (https://github.com/BurntSushi/rust-csv), and what's left is the important last mile of bringing these things together in ergonomic libraries or frameworks.
In Ask HN: What is the state of C++ vs. Rust?
justinsaccount·Jun 8, 2016
This is impressive, but... "A fast reader exploits the capabilities of the storage system"... the graphs show that their storage system is doing 4.00 GB/sec I wonder what processor this is running on and what their storage system this is.. multiple PCIe SSD? I tried running a quick test but only succeeded in OOMing my 8G laptop. Even just doing import paratext it = paratext.load_csv_as_iterator("/dev/shm/tmp/c.log", expand=True, forget=True) x = it.next() Starts eating up all my ram after about a minute of spinning the cpu... so I think they have a slightly different definition of an iterator as everyone else. Compared to cut -d , -f 5 < c.log > /dev/null which runs in a few seconds, or a slightly more domain specific and optimized version of 'cut'[1] that runs even faster (300-500MB/s on a single core depending on which fields
Read the rest
you want) $ du -hs c.log ;wc -l c.log 2.1G c.log 16197412 c.log I also wonder if that is 2.5 GB/s per core. https://github.com/BurntSushi/rust-csv does 241 MB/s in raw mode, so I find it a little hard to believe that this is 10x faster... unless that is while maxing out multiple cores. [1] https://github.com/bro/bro-aux/blob/master/bro-cut/bro-cut.c
In ParaText: CSV parsing at 2.5 GB per second
burntsushi·Feb 22, 2015
It's extremely simple. Even if you don't know Rust, this code should be readable: https://github.com/BurntSushi/rust-csv/blob/master/src/index... Basically, it writes a 64 bit byte offset (big endian) for the start of each record. The very last byte is then the total number of records. (I forget why I'm doing this, since the count should be computable from the size of the index file.) The "start of each record" means "if you point a CSV parser at this byte offset, then it will successfully read a record." So when you want to read record `i`, seek the index to byte offset `i * 8`, read 8 bytes (big endian) into a 64 bit integer, then seek to that byte offset in your CSV data. If the CSV data changes, then you're in trouble. One easy way to handle it (which isn't bulletproof) is to check the last modified times of the index and the CSV file
Read the rest
and regenerate the index (or refuse to use it with an error).
In XSV – A fast CSV toolkit in Rust
burntsushi·Oct 1, 2014
> Rust is one of those languages where I need to work to make the compiler happy, but once I manage that, the code generally works on the first try. I can confirm this. I have a CSV parser[1] that is maybe twice as fast as Python's CSV parser (which is written in C)+. There's nothing magical going on: with Rust, I can expose a safe iterator over fields in a record without allocating. [1] - https://github.com/BurntSushi/rust-csv The docs explain the different access patterns (start with convenience and move toward performance): http://burntsushi.net/rustdoc/csv/#iteratoring-over-records + - Still working on gathering evidence...
In A Fresh Look at Rust
burntsushi·Mar 29, 2014
I've also written a fair bit of Go and have recently taken a full dive into Rust [1, 2] and it's been a ton of fun so far. I really love both languages. Their features and goals have a pretty small intersection, but I enjoy both a lot. If you want to take a dive and are willing to put up with a little instability, now wouldn't be a bad time. The 0.10 release appears to be on the horizon and they just released binary installers, which should make tracking tip much easier: https://mail.mozilla.org/pipermail/rust-dev/2014-March/00922... [1] - https://github.com/BurntSushi/quickcheck [2] - https://github.com/BurntSushi/rust-csv
In Rust vs Go

Showing the 15 most recent of 16 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.