wks-quickstart-firekube

more here than meets the eye and maybe what you said is possible. If you don't need the isolation of a proper VM and were only looking for a roughly VM-shaped system that you CAN "kernel exec" or use nsenter to get processes into, you should look at Footloose[1]. I'm suggesting what you are looking for is actually a container that looks more like a VM or bare-metal machine from the perspective of inits and with the vantage point of the processes running inside. By default, Footloose nodes are running SSH and SystemD, may appear to work similarly to Ignite VMs, but they are Docker containers that may or may not run privileged mode. So, if it suits you, then you could start up Footloose "VMs" as I still call them, strip SSH from them, then nsenter or Docker exec into them as you desire, or run Kubernetes on them and use the Kube API including exec. That is actually a lead in to the next project, known as Firekube[2], kind of a mashup of all these technologies plus one more (wksctl[3]). Firekube integrates both Ignite and Firecracker as well, so you can use it similarly on Linux, (where KVM support is available for Ignite), or MacOS, where Footloose runs container-VMs instead, both behave alike; this suite of projects all put together is a very slick and well integrated package IMHO. It is probably comparable in functionality to Minikube, but with GitOps baked right in. Disclosure: I am not working for Weaveworks, but we are good friends. [1]: https://github.com/weaveworks/footloose [2]: https://github.com/weaveworks/wks-quickstart-firekube/ [3]: https://github.com/weaveworks/wksctl

k8s ecosystem is not as evolved as it will likely soon become is relevant. You may be right that no managed Kubernetes setups will handle your physical servers today, but I think the truth is somewhere between: they're coming / they're already here but most are not quite ready for production / they are here, but I don't know what to recommend strongly. I'm leaning toward the latter (I think that if you wanted a good managed bare metal K8s, you could definitely find it.) I know some solutions that will manage bare metal nodes, but this is not a space I'm intimately familiar with. The solutions that I do know of, are in early enough state of development that I hesitate to mention. It won't be long before this gets much better. The bare metal Cluster API provider is really something, and there are some really amazing solutions being built on top of it. If you want to know where I think this is going, check this out: WKS and the "firekube" demo, a GitOps approach to managing your cluster (yes, even for bare metal nodes) https://github.com/weaveworks/wks-quickstart-firekube I personally don't use this yet, I run kubeadm on a single bare metal node and don't worry about scaling, or the state of the host system, or if it should become corrupted by sysadmin error, or much else really. The abstraction of the Kubernetes API is extremely convenient when you don't have to learn it from scratch anymore, and doubly so if you don't have to worry about managing your cluster. One way to make sure you don't have to worry, is to practice disaster recovery until you get really good at it. If my workloads are containerized, then I will have them in a git repo, and they are disposable (and I can be sure, as they are regularly disposed of, as part of the lifecycle). Make tearing your cluster down and standing it back up a regular part of your maintenance cycles until you're ready to do it in an emergency situation with people watching. It's much easier than it sounds, and it's definitely easier than debugging configuration issues to start over again. The alternative that I would recommend for production right now, if you don't like any managed kubernetes, is to become familiar with the kubeadm manual. It's probably quicker to read it and study for CKA than it would be to canvas the entire landscape of managed providers for the right one. I'm sure it was painful debugging that certificate issue, I have run up against that issue in particular before myself. It was after a full year or more of never upgrading my cluster (shame on me), I had refused to learn RBAC, kept my version pinned at 1.5.2, and at some point after running "kubeadm init" and "kubeadm reset" over and over again it became stable enough (I stopped breaking it) that I didn't need to tear it down anymore, for a whole year. And then a year later certs expired, and I could no longer issue any commands or queries to the control plane, just like yours. Once I realized what was happening, I tried to renew the certs for a few minutes, I honestly didn't know enough to look up the certificate renewal docs, I couldn't figure out how to do it on my own... I still haven't read all the kubeadm docs. But I knew I had practiced disaster recovery well over a dozen times, and I could repeat the workloads on a new cluster with barely any effort (and I'd wind up with new certs.) So I blew the configuration away and started the cluster over (kubeadm reset), reinstalled the workloads, and was back in business less than 30 minutes later. I don't know how I could convince you that it's worth your time to do this, and that's OK (it's not important to me, and if I'm right, in 6 months to a year it won't even really matter anymore, you won't need it.) WKS looks really promising, though admittedly still bleeding edge right now. But as it improves and stabilizes, I will likely use this instead, and soon after that forget everything I ever knew about building kubeadm clusters by hand.