NEneonsunset·Oct 27, 2024
C# has an abstraction for memory-mapped files. You can always use raw pointers and directly call the corresponding OS APIs with interop too.
However, the fastest-performing implementations in 1BRC challenge that were written in C# ended up with inconclusive results whether using memory-mapping over RandomAccess.Read API (which is basically a thin wrapper over read/pread calls) is faster or not: https://github.com/noahfalk/1brc/?tab=readme-ov-file#file-re...
You can relatively easily do 2 GiB/s reads with RandomAccess/FileStream as long as sufficiently large buffer size is used. FileStream default settings already provide a quite good performance, and make it use adaptive buffer size under the hood. Memory-mapping is convenient but it's not a silver bullet (in this context) and page-faulting then mapping the page and filling it with data b
Read the rest
y performing the read within kernel space is not necessarily cheaper than passing a pointer to a buffer to read into.
The challenges in Rust and C# are going to be very similar in this type of task since C# can just pin the GC-allocated arrays to read into, call into malloc or 'stackalloc' the temporary buffer inline, and the rest of implementation will be subject to more or less identical constraints. C# is probably the closest* "high-level" language in feature set to Rust, even if this sounds strange. There's a sibling submission that covers an another angle to this: https://news.ycombinator.com/item?id=41963259
* have not looked through Swift 6 changes in detail yet