3FS

They use 3FS [0] also open sourced [1] [0] https://github.com/deepseek-ai/DeepInfer/blob/main/doc/infer... [1] https://github.com/deepseek-ai/3fs

Should be about 10~20 GiB per session. Save/restore is exactly what DeepSeek does using its 3FS distributed filesystem: https://github.com/deepseek-ai/3fs#3-kvcache With this much cheaper setup backed by disks, they can offer much better caching experience: > Cache construction takes seconds. Once the cache is no longer in use, it will be automatically cleared, usually within a few hours to a few days.

What about the VRAM requirement for KV cache? That may matter more than memory bandwidth. With these GPUs, there are more compute capacity than memory bandwidth than VRAM. DeepSeek got MLA, and then DSA. Qwen got gated delta-net. These inventions allow efficient inference both at home and at scale. If Anthropic got nothing here, then their inference cost can be much higher. DeepSeek also got https://github.com/deepseek-ai/3FS that makes cached reads a lot cheaper with way longer TTL. If Anthropic didn't need to invent and uses some expensive solution like Redis, as indicated by the crappy TTL, then that also contributes to higher inference cost.

Great to see another distributed file system open sourced! It has some interesting design decisions. Have a couple of questions: - How do you go about benchmarking throughput / latency of such a system? Curious if it's different compared to how other distributed filesystems benchmark their systems. - Is network or storage the bottleneck for nodes (at least for throughput)? - From my observations from RDMA-based distributed filesystems, network seems to be the case. - How does the system respond to rand / seq + reads / writes? A lot of systems struggle to scale writes. Does this matter for what workload TernFS is designed for? - Very very interesting to go down the path of writing a kernel module instead of using FUSE or writing a native client in userspace (referring to 3FS [1]) - Any crashes in production? And how do you go about tr

DeepSeek team, which is also an HFT shop, also implemented their DFS - https://github.com/deepseek-ai/3FS

Deepseek has their own and they’re relatively small https://github.com/deepseek-ai/3FS It’s specialized knowledge, hard to do “correctly” (read posix here) but obtainable and implementable by a small team if you pick your battles right. Also supporting very specific use cases helps a lot. It’s also pretty easy to justify as the hardware and software from vanguard tech companies is outrageously expensive. I used to develop software for a blue colored distributed filesystem.

Cool. Also https://github.com/deepseek-ai/3FS by DeepSeek which came out of High-Flyer a Chinese HFT firm.

Foundation is fundamental to iCloud at Apple, and is _something_ at Snowflake, among a few others. Recently DeepSeek used it for https://github.com/deepseek-ai/3FS "The Fire-Flyer File System (3FS) is a high-performance distributed file system designed to address the challenges of AI training and inference workloads." I don't think that there's anything else quite the same, partly because it has some real oddities that manifest because of things like the transaction time limits. At Apple they worked around some of this with https://www.foundationdb.org/files/QuiCK.pdf

>Tectonic some KV store, Tectonic is built on ZippyDB which is a distributed DB built on RocksDB. >What's important here with S3FS is that it supports (1) a fuse client - it just makes life so much easier Tectonic also has a FUSE client built for GenAI workloads on clusters backed by 100% NVMe storage. https://engineering.fb.com/2024/03/12/data-center-engineerin... Personally what stands out to me for 3FS isn't just that it has a FUSE client, but that they made it more of a hybrid of FUSE client and native IO path. You open the file just like normal but once you have a fd you use their native library to do the actual IO. You still need to adapt whatever AI training code to use 3FS natively if you want to avoid FUSE overhead, but now you use your FUSE client for all the metadata operations that the native client would have needed to implem

This has been the case for a while now. 3090 hoarders were always just doing it for street cred or whatever, no way these guys are computing anything of actual value. Tenstorrent is on fire, though. For small businesses this is what matters. If 10M context is not a scam, I think we'll see SmartNIC adoption real soon. I would literally long AMD now because their Xilinx people are probably going to own the space real soon. Infiniband is cool and all, but it's also stupid and their scale-out strategy is non-existent. This is why https://github.com/deepseek-ai/3FS came out but of course nobody had figured it out because they still think LLM's is like, chatbots, or something. I think we're getting to a point where it's a scheduling problem, basically. So you get like like lots of GDDR6 (HBM doesnn't matter anymore) as L0, DDR5 as L1, and NVMe-

https://github.com/deepseek-ai/3FS?tab=readme-ov-file#3-kvca... KVCache is a technique used to optimize the LLM inference process. It avoids redundant computations by caching the key and value vectors of previous tokens in the decoder layers. The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s

Deepseek is already using SSDs for their KV cache: https://github.com/deepseek-ai/3FS

Deepseek is the real "open<something>" that the world needed. Via these three projects, Deepseek has addressed not only efficient AI but also distributed computing: 1. smallpond: https://github.com/deepseek-ai/smallpond 2. 3fs: https://github.com/deepseek-ai/3FS 3. deepep: https://github.com/deepseek-ai/DeepEP

They appear to have Python bindings which seems reasonable from an API / usability perspective? https://github.com/deepseek-ai/smallpond In terms of fast FUSE - also my first question, appears to be`io_uring` + FUSE :) https://github.com/deepseek-ai/3FS/blob/main/src/lib/api/Usr...