hist-rs

>I use nucgen to generate a random 100M line FASTQ file and pipe it into different tools to compare their throughput with hyperfine. This is a strange benchmark [0] -- here is what this random FASTQ looks like: $ nucgen -n 100000000 -l 20 | head -n8 >seq.0 TGGGGTAAATTGACAGTTGG >seq.1 CTTCTGCTTATCGCCATGGC >seq.2 AGCCATCGATTATATAGACA >seq.3 ATACCCTAGGAGCTTGCGCA There are going to be very few [*] repeated strings in this 100M line file, since each >seq.X will be unique and there are roughly a trillion random 4-letter (ACGT) strings of length 20. So this is really assessing the performance of how well a hashtable can deal with reallocating after being overloaded. I did not have enough RAM to run a 100M line benchmark, but the following simple `awk` command performed ~15x faster on a 10M line benchmark (using the same hyp

Was sitting around in meetings yesterday and remembered an old shell script I had to count the number of unique lines in a file. Gave it a shot in rust and with a little bit of (over-engineering)™ I managed to get 25x throughput over the naive approach using coreutils as well as improve over some existing tools. Some notes on the improvements: 1. using csv (serde) for writing leads to some big gains 2. arena allocation of incoming keys + storing references in the hashmap instead of storing owned values heavily reduced the number of allocations and improves cache efficiency (I'm guessing, I did not measure). There are some regex functionalities and some table filtering built in as well. happy hacking

Was sitting around in meetings today and remembered an old shell script I had to count the number of unique lines in a file. Gave it a shot in rust and with a little bit of (over-engineering)™ I managed to get 25x throughput over the naive approach using coreutils as well as improve over some existing tools. Some notes on the improvements: 1. using csv (serde) for writing leads to some big gains 2. arena allocation of incoming keys + storing references in the hashmap instead of storing owned values heavily reduced the number of allocations and improves cache efficiency (I'm guessing, I did not measure). There are some regex functionalities and some table filtering built in as well. happy hacking