TAtastyminerals2·Apr 22, 2024
If you need just csv/tsv parsing, you can also take a look at https://github.com/eBay/tsv-utils
In Frawk: An efficient Awk-like programming language. (2021)eBay
tsv-utils
D2016–2024
eBay / tsv-utils
tsv-utils
eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.
17Distinct people
20Mentions
0Explicit recommenders
1.5KGitHub stars
What to know before exploring
What it is
eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- burntsushi/xsv · linked in 5 of the same discussions
- batchlabs/charlatan · linked in 3 of the same discussions
- dbohdan/sqawk · linked in 3 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year20 dated mentions
Observed mentions: 2016-04-13 18:02:09 UTC to 2024-04-22 16:25:45 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 6 | 1 | 5 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 4 | 0 | 4 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
20 sourcesYou might want to look at tsv-utils, or a similar project: https://github.com/eBay/tsv-utils
For the SQL part, but maybe a lot heavier, you can use one of the projects listed on this page: https://github.com/multiprocessio/dsq (No longer maintained, but has links to lots of other projects)
In Tracking SQLite Database Changes in GitNice article. I'm left wondering:
- Are there any benchmarks available?
- Is any full code example available (as opposed to a snippet)?
- Does it handle real-world edge cases (unix vs dos line ends, non-"normal" quoting etc)
- Does it normalize (e.g. `"hi"` and `hi` are both returned as `hi`)?
- How does it compare, performance-wise (or in other ways) with https://github.com/BurntSushi/xsv, https://github.com/eBay/tsv-utils or https://github.com/liquidaty/zsv? (disclaimer: I wrote zsv)
In Splitting CSV files at 3GB/sFor anything down and dirty, what's wrong with -F'"'? For anything fancy there are plenty of things like the below.
eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.
includes csv to tsv: https://github.com/eBay/tsv-utils
HT: https://simonwillison.net/
In Modernizing AWK, a 45-year old language, by adding CSV supportWhen you have "format wars", the best idea is usually to have a converter program change to the easiest to work with format - unless this incurs a space explosion as per some image/video formats.
With CSV-like data, bulk conversion from quoted-escaped RFC4180 CSV to a simpler-to-parse format is the best plan for several reasons. First, it may "catch on", help Microsoft/R/whoever embrace the format and in doing so squash many bugs written by "data analyst/scientist coders". Second, in a shell "a|b" runs programs a & b in parallel on multi-core and allow things like csv2x|head -n10000|b or popen("csv2x foo.csv"). Third, bulk conversion to a random access file where literal delimiters cannot occur as non-delimiters allows trivial file segmentation to be nCores times faster (under often satisfied assumptions). There are some D tools for t
Read the rest
his bulk convert in https://github.com/eBay/tsv-utils and a much smaller stand-alone Nim tool https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim . Optional quoting was always going to be a PITA due to its non-locality. What if there is no quote anywhere? Fourth, by using a program as the unit of modularity in this case, you make things programming language agnostic. Someone could go to town and write a pure SIMD/AVX512 converter in assembly even and solve the problem "once and for all" on a given CPU. The problem is actually just simple enough that this smells possible.
I am unaware of any "document" that "standardizes" this escaped/lossless TSV format. { Maybe call it "DSV" for delimiter separated values where "delimiters actually separate"? Ironically redundant. ;-) } Someone want to write an RFC or point to one? It can be just as "general/lossless" (see https://news.ycombinator.com/item?id=31352170).
Of course, if you are going to do a lot of data processing against some data, it is even better to parse all the way to down to binary so that you never have to parse again (well, unless you call CPUs loading registers "parsing") which is what database systems have been doing since the 1960s.
I love these TSV utilities: https://github.com/eBay/tsv-utils Granted they're for "T"sv files, not "C"sv, but there's a handy `csv2tsv` utility included.
There's also `xsv`, from the author of `ripgrep`: https://github.com/BurntSushi/xsv
In Awk: `Begin { ` Part 1eBay's tsv-utils author here. A fair bit of performance benchmarking was done on the tools with the goal of exploring this type of use (data crunching). D did really well (tsv-utils are fast!). There's more info on the benchmarks page in the github repo. Perhaps the best summary is the slides from a talk I did at 2018 DConf. Links:
- tsv-utils repo: https://github.com/eBay/tsv-utils
- Performance studies: https://github.com/eBay/tsv-utils/blob/master/docs/Performan...
- Talk slides: https://github.com/eBay/tsv-utils/blob/master/docs/dconf2018...
In Specification for the D Programming LanguageIt doesn't even need to be an alternative to Python or R, since it's really easy to interoperate with both languages. This is in addition to tools like eBay's tsv-utils.
http://code.dlang.org/packages/pyd
https://embedr.netlify.com/
http://code.dlang.org/packages/autowrap
https://github.com/eBay/tsv-utils
In Specification for the D Programming LanguageAlso along these lines is tsv-utils:
https://github.com/eBay/tsv-utils/blob/master/README.md
There are benchmark results comparing tsv-utils to a variety of similar tools including xsv.
In Doing a database join with CSV filesA plug of my tools:
To visualize data coming in from a pipe, can pipe it to
https://github.com/dkogan/feedgnuplot
Very useful in conjunction with other tools to provide filtering and manipulation. For instance (the first one is mine):
https://github.com/dkogan/vnlog
https://www.gnu.org/software/datamash/
https://csvkit.readthedocs.io/
https://github.com/johnkerl/miller
https://github.com/eBay/tsv-utils-dlang
http://harelba.github.io/q/
https://github.com/BatchLabs/charlatan
https://github.com/dinedal/textql
https://github.com/BurntSushi/xsv
https://github.com/dbohdan/sqawk
https://stedolan.github.io/jq/
https://github.com/benbernard/RecordStream
In An introduction to data processing on the Linux command lineYou can use EBay tools as well, such as https://github.com/eBay/tsv-utils after importing a CSV file as TSV.
In How To Open and Manipulate Large CSV Files On A MacCool that there are some specialty tools in this space and will use if I hit that performance challenge.
Minor nit, I found the performance table (https://github.com/eBay/tsv-utils/blob/master/docs/Performan...) confusing at first and second glance. Alternating colors indicate.. different OSes? That doesn't seem to be the important message to convey as you are trying to show the speed of your tool and not the OS. Recommend to use the coloration to provide differentiation between tools instead of OS.
In TSV Utilities: Command line tools for large, tabular data filesThere're lots of tools in this space that are similar in a very general way, but have widely different design choices. My tookit is
https://github.com/dkogan/vnlog
It's tsv-utils-like, but is strictly a wrapper around existing tools. So filtering and transformations are interpreted literally as awk (or perl) expressions. And the various cmdline options match the standard tool options because they ARE the standard tools. So you get a very friendly learning curve, but something like tsv-utils is probably faster and probably more powerful. And it looks like tsv-utils references fields by number instead of by name. Many of the others (mine included) use the field names, which makes a MAJOR usability improvement.
Other tools in no particular order:
https://csvkit.readthedocs.io/
https://github.com/johnkerl/miller
https://github.com/eBay/tsv-ut
Read the rest
TSV Utilities: Command line tools for large, tabular data files
467 points · Open discussionThis is becoming a really crowded space. Some other similar tools that make slightly different design choices and that have variable envisioned use cases:
- https://github.com/dkogan/vnlog
- https://csvkit.readthedocs.io/
- https://github.com/johnkerl/miller
- https://github.com/BurntSushi/xsv
- https://github.com/eBay/tsv-utils-dlang
- https://stedolan.github.io/jq/
- http://harelba.github.io/q/
- https://github.com/BatchLabs/charlatan
- https://github.com/dinedal/textql
- https://github.com/dbohdan/sqawk
(disclaimer: vnlog is my tool)
In Show HN: UXY – adding structure to Unix toolsShowing the 15 most recent of 20 sources. Enable JavaScript to search and browse every indexed mention.