zfs-localpv

se case doesn't justify k8s in the first place, I'd stick to what already works ;)

rust, iSCSI/NVMe-oF based, bring your own lower layer) - zfs-localpv[3] Funnily enough, I've been really interested in zfs and btrfs lately so have been learning a lot about the field, but cStor didn't work when I first ran it. I know "written in rust" has become somewhat of a buzzword, but don't forget that it has actual meaning -- it means that they will be able to get the raw performance possible without a language runtime or garbage collection to hold them back. This is critical for "systems" programming, and is the domain rust was built for. When I see rust, I basically think -- this could be as fast as average C (and safer) if you get even just pretty good developers. Also a note on this -- one of the things that's nice about OpenEBS's offerings is that writes are synchronous. This is obviously a problem[4] for performance of course, but it's a massive relief for building resiliency into systems. I'm not a classical sysadmin, but it puts me at ease that I can give up some throughput to know that if one of my servers goes completely kaput, the other can pick up at the last write on the other node, with zero guesswork on my end. The switchover is of course a little hairy but this is what OpenEBS smoothes out for you. For this not to be completely an ad for OpenEBS, I do want to note that the other prominent option that scales from hobbyist to enterprise (and arguably more so) is Rook[5] which manages Rook clusters. Rook the organization seems to be branching out to position themselves as Cassandra storage and a bunch of other stuff, but the key thing they brought to the table was a mostly hands-off operator for operating Ceph[6] which is certified enterprise grade(tm) FREE software. Ceph is basically the linux of the storage appliance world. Ceph (via Rook) is another really good choice for getting storage up and running on a kubernetes (or whatever else) because it offers similar benefits to OpenEBS's solutions but does it slightly different. Instead of container-attached-storage, it solves the features come at the filesystem layer-- you get a tunable "ceph" installation (you should definitely choose BlueStore btw) which gives you: - replication (like the iSCSI approaches of OpenEBS) - checksumming (like ZFS and zfs-backed OpenEBS)[7] - striping (like ZFS/btrfs and zfs-backed OpenEBS depending on setup) - compression (like ZFS)[7] Anyway, I don't run any servers that use NVMe but nice to see this article, hope the people at OpenEBS are still doing well, love their entry into the space. [EDIT] - I totally forgot and should mention, you can obviously always bring your own lower layer, ex creating a xfs-formatted ZVOL on top of ZFS on the host and then exposing that drive to whatever. Excellent example is a recent talk from Japan which I've mentioned before[8][9] which uses this to run Ceph on ZFS. [0]: https://docs.openebs.io/docs/next/casengines.html [1]: https://docs.openebs.io/docs/next/jiva.html [2]: https://docs.openebs.io/docs/next/alphafeatures.html#mayasto... [3]: https://github.com/openebs/zfs-localpv/ [4]: https://github.com/longhorn/longhorn/issues/1242#issuecommen... [5]: https://rook.io/docs/rook/ [6]: https://docs.ceph.com/en/latest/architecture/ [7]: https://ceph.io/community/new-luminous-bluestore/ [8]: https://speakerdeck.com/takutakahashi/ceph-on-zfs [9]: https://news.ycombinator.com/item?id=24784596

sume - Add Ceph via Rook[1] to get synchronous remote replication on every write (if you want it), and give ceph the drives you made (the zvol(s)) - Expose storage to your kubernetes cluster that is replicated (depending on how you set up ceph) and easy to back up and/or duplicate at the node level (w/ zfs send). You can back up at multiple levels, nodes themselves (with some replication lag since zfs doesn't so sync replication) and the workloads that run on them (which are sync replicated across the network). There is some overlap in these tools -- ceph's relatively new bluestore engine does checksumming now and self-heals, so there's not so much reason to use ZFS at the lowest level at all, but ZFS does have some features that ceph doesn't, and is somewhat easier to manage with less moving parts in user space. [0]: https://github.com/openebs/zfs-localpv/ [1]: https://github.com/rook/