csvquote

Good idea! Looks similar to something I wrote called csvquote https://github.com/dbro/csvquote , which enables awk and other command line text tools to work with CSV data that contains embedded commas and newlines.

I work in data processing and I use awk occasionally to work with csv files often gigabytes in size. I join csv files that each have a header with awk '(NR == 1) || (FNR > 1)' *.csv > joined.csv Note this only works if your csv files don't contain new lines. However if they do, I recommend using https://github.com/dbro/csvquote to circumvent the issue. Yesterday I used awk as a QA tool. I had to subtract a sum of values in the last column of one csv file from another, and I produced a expr $(tail -n+2 file1.csv | awk -F, '{s+=$(NF)} END {print s}') - $(tail -n+2 file2.csv | awk -F, '{s+=$(NF)} END {print s}') beauty. This allowed me to quickly check whether my computation was correct. Doing same in pandas would require loading both files into RAM and writing more code. However I avoid writing awk programs that are longer than a few

To simplify working with CSV data using command line tools, I wrote csvquote ( https://github.com/dbro/csvquote ). There are some examples on that page that show how it works with awk, cut, sed, etc.