lakeFS

The premise is that Git-LFS sucks, so we need to build a new data versioning system (in Rust, from scratch). While I mostly agree with this premise, but there are already lots of existing (mature) data versioning systems with the same tricks under the hood: - Pachyderm (Go): https://github.com/pachyderm/pachyderm - XetHub (acquired by HuggingFace): https://huggingface.co/blog/xethub-joins-hf - LakeFS (Go): https://github.com/treeverse/lakeFS - Oxen (Rust): https://github.com/Oxen-AI/Oxen I guess with AI, anyone can vibe code a content-addressed, chunk-level deduped, versioning system in Rust these days... But jokes aside, Lore seems really cool! What's interesting is the realization that different domains/industries have similar problems, but they don't seem to be cross-polinating. In this case AI and Gaming both need a storage system th

That's fair. What makes this unique is the versioned, composable filesystem. It's built on top of lakeFS (https://github.com/treeverse/lakeFS) so it scales really well, unlike other solutions that try and do this with Git directly.

Fair enough - the underlying technology is indeed open source (https://github.com/treeverse/lakeFS) - the service provides the hosting and tooling to make it easy for consumption by agents.

Might want to look at purpose built tools for that such as lakeFS (https://github.com/treeverse/lakeFS/) * Disclaimer: I'm one of the creators/maintainers of the project.

Might want to check out lakeFS: https://github.com/treeverse/lakeFS (full disclosure: I'm one of the creators)

Check out lakeFS (https://github.com/treeverse/lakeFS). It doesn't rely on the object store for ordering and is highly influenced by Git itself (but designed to work on object stores at very large scales)