MAmagicalhippo·Feb 3, 2024Recommendation phrase
Unfortunately I have not.
The core concept is rather simple though: iSCSI allows for an iSCSI server to expose raw block devices (called targets[1]), so do that and use them from a client machine as-if they were locally connected disks.
So I used LIO[2] as the iSCSI server on each of the Pi's, making sure to use the "by-id" to reference the block device so it would work fine across reboots (skipped changing "directories"):
sudo targetcli
/backstores/block> create name=block0 dev=/dev/disk/by-id/usb-SAMSUNG_MZ7PD256HAFV-000_0123456789000000005-0:0
/iscsi> create
/iscsi> cd iqn.2003-01.org.linux-iscsi.rpi4.armv7l:sn.fc1b1c9879a1/
/iscsi/iqn.20....fc1b1c9879a1> cd tpg1/luns
/iscsi/iqn.20...9a1/tpg1/luns> create /backstores/iblock/block0
Note you might have to use /block/ instead of /iblock/ in the last command there[
Read the rest
3], depending on targetcli version.
Then you create the portal which exposes the target (backed by the raw disk) to the network as normal (see Debian guide fex). I did it like this, you have to adjust for the name of your device:
/iscsi/iqn.20...1c9879a1/tpg1> cd portals
/iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 3260
/iscsi/iqn.20.../tpg1/portals> create 10.1.1.101
/iscsi/iqn.20.../tpg1/portals> cd ..
/iscsi/iqn.20...1c9879a1/tpg1> set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
This creates a portal which exposes a single LUN, backed by the raw disk, on the IP 10.1.1.101, and disables authentication and demo mode (write protection).
Next you install the iSCSI client (called initiator)[4], I did this on a separate machine. There I connected to each of the target on each Pi. Once successful you should get a number of disks under /dev/disk/by-path/ip-*, which you can then specify when importing the pool on the iSCSI client machine, say
zpool -d /dev/disk/by-path/ip-X -d /dev/disk/by-path/ip-Y
Since you're exposing the raw block device, there's no difference in that regard to having the disk plugged into the client machine directly.
Btw, I recommend not messing with authentication at first, though it's not terribly difficult to set up.
edit: As mentioned the nice thing about this approach is that it's effectively "harmless". If it doesn't work out for you, you can always put all the disks in a single machine and import the pool as normal.
Also, iSCSI has a lot of robustness built in. For example the iSCSI client will temporarily store writes and re-issue once the target (server) is back online. I safely rebooted one of the Pis while copying data to the pool, for example.
[1]: https://en.wikipedia.org/wiki/ISCSI#Concepts
[2]: https://wiki.debian.org/SAN/iSCSI/LIO
[3]: https://github.com/ClusterLabs/resource-agents/pull/1373
[4]: https://wiki.debian.org/SAN/iSCSI/open-iscsi