flocker

Clustered filesystems like Ceph or GlusterFS might help with the reliability part, but latency and bandwidth still remain a problem. The other thing I was quite interested in at one point was flocker, a volume manager which is responsible for replicating/migrating data along with the container it is connected to. The company shut down quite a while ago but code is still available: https://github.com/ClusterHQ/flocker

If anybody's wondering about the portable volumes, they're based on Flocker (https://github.com/ClusterHQ/flocker) which uses ZFS to move data around between nodes.

Hey, Luke here, CTO at ClusterHQ. As well as being proud of this company milestone, I can't wait to get powerstrip (https://github.com/clusterhq/powerstrip) and flocker (https://github.com/clusterhq/flocker) working together so that we can start composing flocker with other docker ecosystem tools. It will be a great day that you can have a docker swarm with flocker and weave. Or kubernetes with flocker and socketplane. Or even mesosphere with rancher and calico ;) We've got some pretty neat demos of powerstrip prototypes up here, fwiw: http://clusterhq.com/blog/powerstrip-prototype-docker-extens... Any questions just shout! Cheers, Luke

We are working on solving this specific problem with Flocker at https://github.com/clusterhq/flocker/ - see http://docs.clusterhq.com/en/latest/introduction.html for a description of how data migration works. We will be working on high availability after we've made some more progress on getting to a production-ready framework for volume management. We are super interested in building this as a plugin for Swarm, and are trying to work with Docker to make sure the plugins system is suitable for us to be able to contribute it for Swarm users in general.

https://github.com/clusterhq/flocker seems to be doing some interesting stuff in that direction.

ZFS is also being worked on as a Docker storage layer https://github.com/ClusterHQ/flocker

You can use Gentoo Prefix for this: https://www.gentoo.org/proj/en/gentoo-alt/prefix/ It builds everyting except libc. Support for building libc is under development by the project. I used an early version of it to provide a build environment for the ZFSOnLinux kernel modules on CoreOS: https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tuto...

Yes: https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tuto...

I am working on ZFS support for CoreOS. A snapshot of my WIP proof of concept was posted by my employer yesterday: https://github.com/ClusterHQ/flocker/blob/zfs-on-coreos-tuto... CoreOS uses btrfs as its rootfs. I imagine that the CoreOS developers managed to avoid issues like ENOSPC by virtue of not writing to their rootfs very much. I did not have that luxury since I compiled a Gentoo GNU userland on top of it during the course of development. I encountered numerous ENOSPC errors on btrfs when developing the ZFS port to CoreOS and even hit ENOSPC errors when trying to correct the btrfs ENOSPC errors with `btrfs balance /`. I would not consider btrfs ready for production use, but your mileage will vary.

At the expensive of sounding like I'm just plugging my own company, this is what we're working on in the open-source project Flocker (https://github.com/ClusterHQ/flocker). We think that data-services like databases, queues and key-value stores, and anything else with state should be able to run inside docker containers too. Yes, you can already run a database in a docker container, but from an ops perspective, this is a nightmare and very far from what you want want to be able to do in a production system. Would love your feedback on what we're building and even more for you to get involved. Flocker is licensed under Apache 2.0, so feel free to get involved.

Check Flocker https://github.com/ClusterHQ/flocker

Cool! If you are interested in docker and databases, my company is working on an oss project to simplify the deployment and migration of databases and their volumes: https://github.com/ClusterHQ/flocker It's aimed more at production use cases where things like moving volumes around for db upgrades are a requirement, but you might find it useful. Its open source so would love your feedback and involvement.