marmot

Working on Marmot https://github.com/maxpert/marmot recently added support for vector index. My local benchmarks show pretty decent QPS with less than GB of RSS on DBpedia dataset. Interesting part is that I started off implementing a research paper for indexing and performance was not good enough. I ended up tuning things up for my own use-case and ended up with good enough replicatable RAG store.

I am as usual working on Marmot https://github.com/maxpert/marmot I've got replicas now working with DML proxy. This essentially means I can now have a cluster of primaries, and then spin up replicas on demand and nodes talking to local host will never see their mutation work pretty transparently from readonly-replicas. While PoC works now the snapshot restore is extremely inefficient IMO yet.

Kind of what I've been working on to build tenancy on top of SQLite CDC to make it a simple repayable SQLite for Marmot (https://github.com/maxpert/marmot). I personally think we have a synergy here, would drop by your discord.

Disclaimer: I am author of Marmot https://github.com/maxpert/marmot so I will sound extremely bias. I have been asked multiple times on why I chose SQLite and not Turso. I've always responded people that I don't trust an open-source project once it's backed by a VC firm. I've moved away from Redis to Val-Key for same reason, and we have seen the Redis train-wreck in slow-mo. I hope at no point in future Turso ever ends up in that state, but chances are pretty high. At this point the "compatible with SQLite" has become a marketing term IMO, we all know how easy it is to break compatibility here or SQLite to break compatibility.

LOL what are the odds, I posted in `Show HN` about Marmot today https://github.com/maxpert/marmot/releases/tag/v2.2.0 and in my head I was thinking exact same thing for supporting MySQL's JSON datatype. At some level I am starting to feel, I might as well be able to expose a full MongoDB compatible protocol that let's you talk to tables as collections, solving this problem once it for all! But this technique I guess is very common now.

Marmot hits major milestone I've been solo pushing (with help of AI), to really mature out the system. And yesterday I have hit the first milestone where MySQL API is stable enough to make WordPress install and work flawlessly. System is now in a place where you can spin off a cluster, and then spin off many replicas you want to scale out. I wanted to build something that is actually hard with AI, and it's been long and tough journey trying various AI tools, and huge amount of learning. I will follow up with a blog post, but since I have all the scripts to help you guys spin up cluster and examples I wanted to put it out there so that you guys can poke holes and help me improve it. Source is available at: https://github.com/maxpert/marmot/ Most fun part is a Wordpress running in cluster: https://github.com/maxpert/marmot/tree/master/exam

there's various options now, but im most interested in Marmot - which is multi-master and just came out of a 2 year hibernation with a big overhaul that introduced a native gossip mechanism to replace NATS/Raft https://github.com/maxpert/marmot/

Not to be confused with Marmot, the multi-master distributed SQLite server, which has been around for a couple years longer and just came out of 2 years in hibernation, shed its NATS/Raft fat in favour of a native gossip protocol for replication. https://github.com/maxpert/marmot

I use marmot which gives me a multi master SQLite. It uses a simple crdt table structure, and allows me to have many live SQLite instances in all data enters. A Nat Jetstream server is used as a core with all SQLite DBS connected to it. It operates off the WAL and is simple to install. It also replicates all blobs to S3 , with the directory structure in the SQLite db. With a Cloudflare domain , the users request is automatically sent to the nearest Db. So it replaces cloudscapes D1 system for free . Just a hetzner 4 euro cos is enough. Mine is holding about 200 gb of data. https://github.com/maxpert/marmot

If you're willing to accept eventual consistency (a big ask, but acceptable in some scenarios) then there are options like marmot [1] that replicate cdc over nats. [1]: https://github.com/maxpert/marmot

https://github.com/maxpert/marmot is SQLite replication thats eventually consistent and doesnt require a master server.

I've written a whole SQLite replication system that works on top of RAFT ( https://github.com/maxpert/marmot ). Best part is RAFT has a well understood and strong library ecosystem as well. I started of with libraries and when I noticed I am reimplementing distributed streams, I just took off the shelf implementation (https://docs.nats.io/nats-concepts/jetstream) and embedded it in system. I love the simplicity and reasoning that comes with RAFT. However I am playing with epaxos these days (https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf), because then I can truly decentralize the implementation for truly masterless implementation. Right now I've added sharding mechanism on various streams so that in high load cases masters can be distributed across nodes too.