1brc

There are some good ideas for this type of problem here: https://github.com/dannyvankooten/1brc After you deal with parsing and hashes, basically you are IO limited, so mmap helps. The C code takes less than 1.4s without any CUDA access. Because there is no compute to speak of, other than parsing and a hashmap, a reasonable guess is that even for the optimal CUDA implementation, the starting of kernels and transfer of data to the GPU would likely add a noticeable bottleneck and make the optimal CUDA code slower than this pure C code.

c dominates every other language again...https://github.com/dannyvankooten/1brc#submitting

Looks like the C version is probably more than twice as fast compared to the .NET and JDK versions (which are now close)[1]. Someone needs to do the full modern hardware treatment (for ex. SIMD) to the C version (perhaps even adding some ASM) to see where we land. [1] https://github.com/dannyvankooten/1brc

There’s an implementation in C which should run in well under 2 seconds on your M3. https://github.com/dannyvankooten/1brc

You can run the bin/create-sample program from this C implementation here: https://github.com/dannyvankooten/1brc It’s just the city names + averages from the official repository using a normal distribution to generate 1B random rows.

There are a handful of implementations in other languages already. Here’s mine in C99: https://github.com/dannyvankooten/1brc I’ve also seen versions in Rust, Go, Python, Clickhouse and DuckDB. The discussions tab on the GitHub repo lists some of these.

Very fun challenge that nerd sniped me right away. Had to do a C version in standard C99 with POSIX threads. It[1] clocks in at just under 4 seconds on my AMD Ryzen 4800U Laptop CPU. Should run about 10-20% faster than that on the mentioned Hetzner hardware. - Since we only do one decimal of floating point precision it uses integer math right from the get-go. - FNV1-a hash with linear probing and a load factor well under 0.5. - Data file is mmap’d into memory. - Data is processed in 8 totally separate chunks (no concurrent data structures) and then those aggregations are in turn aggregated when all threads have finished. 1: https://github.com/dannyvankooten/1brc