FASTER

I get excited every time I see a paper from Bradish. I've learned so much about high performance software from studying systems that he has worked on. (not to diminish his many co-authors and contributors) Some of his other projects: [0] https://github.com/microsoft/garnet [1] https://github.com/microsoft/FASTER [2] https://github.com/microsoft/Trill

Reading, 'ensuring that data is actually written and stays written is rather difficult', immediately reminded me of https://github.com/microsoft/FASTER (its not written in Go though), which is basically dealing with just that outlet ( except I think the KV store might be ram heavy, been a bit since I last looked at it )

You would be surprised by performance of modern .NET :) Writing no-alloc is oftentimes done by reducing complexity and not doing "stupid" tricks that work against JIT and CoreLib features. For databases specifically, .NET might actually be positioned very well with its low-level features (intrisics incl. SIMD, FFI, struct generics though not entirely low-level) and high-throughput GC. Interesting example of this applied in practice is Garnet[0]/FASTER[1]. Keep in mind that its codebase still has many instances of un-idiomatic C# and you can do way better by further simplification, but it already does the job well enough. [0] https://github.com/microsoft/garnet [1] https://github.com/microsoft/FASTER

For something like this it would make sense to leverage existing libraries instead of "rolling your own". For the storage engine especially, it's very hard to get both performance and safety right. Building on top of Microsoft's FASTER[1] Log and FASTER KV would be a good option for a storage engine, as that is also C# and notably provides higher performance than the in-memory ConcurrentDictionary class in the standard library! LiteDB uses locks for writers, which means it has virtually no chance of being anywhere near as performant as a lock-free log. [1] https://github.com/microsoft/FASTER

I think I saw an article saying it could get 250m, but here I’m seeing Microsoft recorded it up to about 160m/second. It’s way better for read heavy stuff, and there are some trade offs. It’s written in C++ and another clone in C# from them https://microsoft.github.io/FASTER/ https://github.com/Microsoft/FASTER/wiki/Performance-of-FAST... https://microsoft.github.io/FASTER/

A vaguely similar project that might be of interest is: https://github.com/microsoft/FASTER It's also an "unbundled" low-level component that one could use as the foundation for a database engine or whatever. According to Microsoft, FASTER is not just "fast", but significantly faster than even some basic in-memory data structures that ship in the .NET standard library! The downside is that it doesn't (yet) support some more advanced features like multi-server distributed mode. However, that relative simplicity may be preferred in some scenarios...

Last time i looked into it there weren't that many i could find. There is https://github.com/tikv/tikv which uses rocksdb with raft. and there is faster https://github.com/microsoft/FASTER/ .

FTA, https://github.com/Microsoft/FASTER/wiki/Performance-of-FAST...

src is in native https://github.com/Microsoft/FASTER/tree/master/cs/src/nativ... Looks like some file IO functions and __rdtsc

Here is the relevant metadata: https://github.com/Microsoft/FASTER/blob/master/cs/src/core/... Should be usable in NetCoreApp 2.0 and up, or .NET full framework 4.6 and up. In other words, yes, it will work on recent versions of NetCore and classic .Net. However I also see a .dll file in use https://github.com/Microsoft/FASTER/blob/master/cs/src/core/... Which suggests that this won't work in NetCoreApp on linux. It's not crossplatform unless it eliminates this or supplies linux .so binaries and so on.

This. Somebody already filled an issue https://github.com/Microsoft/FASTER/issues/2

Interestingly, the hash bucket structure presented in their papers does not match the one on their code [0], but that probably means they've made some sort of improvement post-publications. [0] https://github.com/Microsoft/FASTER/blob/master/cc/src/core/...