talos

There's systems like Talos where you are pretty much explicitly not allowed to touch Linux, where essentially the only thing the computer presents that you can control is Kubernetes itself. At that point, the OS is kubernetes basically; I find it much harder to entertain pretenses that it's not when you get that far along. https://github.com/siderolabs/talos But it's really a gradient. As more and more services and systems move from being systemd services (or whatever) to in kubernetes, I see it as the position along a gradient shifting. Both are operating systems. They're different operating systems, doing different things, but both manage and provide resources for user payloads, both run apps, both host a user interface (apiserver for kubernetes), both have state (also apiserver, vs file-systems).

Many other distros almost work out of the box (as in they boot and run without any modifications). The custom kernel patches you mentioned address mainly non-critical bugs such as a bug where the Realtek r8127 stops working after a reboot (but it works if you turn it off and on again) [1] [2]. I'd consider it in a way better state than trying to run other Linux distros on certain device tree-based devices like, say, Qualcomm Snapdragon machines. The regular NVIDIA drivers with the open source kernel modules work just fine. Talos Linux supports DGX Spark since version 1.12 [3]. I also know of people using Fedora and nixOS successfully. [1] https://github.com/NVIDIA/NV-Kernels/compare/ea55925ab430f1e... [2] https://forums.developer.nvidia.com/t/realtek-r8127-ethernet... [3] https://github.com/siderolabs/talos/issues/12170

talos is already in use by https://github.com/siderolabs/talos, was confused for a second when I saw talos and wasm for a second, got excited about native wasm pod support.

Talos Linux [1], "the Kubernetes Operating System", is written in Go. That means it exactly works as the little demo here, where the Kernel hands over to a statically compiled Go code as init script. Talos is really an interesting linux distribution because it has no classical user space, i.e. there is no such thing as a $PATH including /bin, /usr/bin, etc. The shell is instead a network API, following the kubernetes configuration-as-code paradigm. The linux host (node) is supposed to run containerized applications. If you really want to, you can use a special container to get access to the actual user space from the node. [1] https://www.talos.dev/ [2] https://github.com/siderolabs/talos/releases/tag/v1.11.5

I'm also one of the maintainers of stagex. fwiw, Talos now uses stagex for their builds: https://github.com/siderolabs/talos/releases/tag/v1.10.0-alp...

TIL about Talos (https://github.com/siderolabs/talos, via your github/onedr0p/cluster-template link). I'd been previously running k3s cluster on a mixture of x86 and ARM (RPi) nodes, and frankly it was a bit of a PiTA to maintain.

I considered deploying Talos a few weeks ago, and I ran into this: https://github.com/siderolabs/talos/issues/8367 Unless I’ve missed something, this isn’t a big deal in an AWS-style cloud where extra storage volumes (EBS, etc) have essentially no incremental cost, and maybe it’s okay on bare metal if the bare metal is explicitly designed with a completely separate boot disk (this includes Raspberry Pi using SD for boot and some other device for actual storage), but it seemed like a mostly showstopping issue for an average server that was specced with the intent to boot off a partition. I suppose one could fudge it with NVMe namespaces if the hardware cooperates. (I’ve never personally tried setting up a nontrivial namespace setup.) Has anyone set up Talos in a useful way on a server with a single disk or a single RAID array?

Super cool. I always enjoy reading about systems that challenge, well, "ossified" assumptions. An OS not providing a shell, for example? Madness! ... or is it genius, if the OS has a specific purpose...? It's thought-provoking, if nothing else. I'm a bit skeptical of parts. For instance, the "init" binary being less than 400 lines of golang - wow! And sure, main.go [1] is less than 400 lines and very readable. Then you squint at the list of imported packages, or look to the left at the directory list and realize main.go isn't nearly the entire init binary. That `talosctl list` invocation [2] didn't escape my notice either. Sure, the base OS may have only a handful of binaries - how many of those traditional utilities have been stuffed into the API server? Not that I disagree with the approach! I think every company eventually replaces dir

Talos Linux basically implements their entire userspace in Go and its similar to BottleRocketOS, because it is designed to host Kubernetes. https://www.talos.dev/ https://github.com/siderolabs/talos

You might be surprised to find that Talos os (linux distro for kubernetes) mostly uses Go: https://github.com/siderolabs/talos

I've been using a 3 nuc (actually Ryzen devices) k3s on SuSE MicroOS https://microos.opensuse.org/ for my homelab for a while, and I really like it. They made some really nice decisions on which parts of k8s to trim down and which Networking / LB / Ingress to use. The option to use sqlite in place of etcd on an even lighter single node setup makes it super interesting for even lighter weight homelab container environment setups. I even use it with Longhorn https://longhorn.io/ for shared block storage on the mini cluster. If anyone uses it with MicroOS, just make sure you switch to kured https://kured.dev/ for the transactional-updates reboot method. I'd love to compare it against Talos https://www.talos.dev/ but Talos's lack of support for a persistent storage partition (only separate storage device https://github.com/siderolabs/talos/is

To add to the other excellent answers, I would recommend adding Bottlerocket to your reading list: https://github.com/bottlerocket-os/bottlerocket#readme I'm also aware of (but haven't used) https://github.com/siderolabs/talos#readme I just realized your question may have implied a desktop os, whereas Bottlerocket, Flatcar, and likely the others in this specific thread are server-side. I don't have much experience with trying to solve that problem on the desktop except for the horror-show that is snap

The amount and variety of machine images shipped is honestly impressive: https://github.com/siderolabs/talos/releases/tag/v1.0.6 First time I have seen a project publish vmware-arm64.ova for ESXi arm edition. Is it still possible to exec into a shell on a cluster node via something like https://github.com/kvaps/kubectl-node-shell ?

For a project similar to bottlerocket, checkout https://github.com/talos-systems/talos. It is geared for cloud, VMware, and bare metal users. We have integrations with Cluster API for each, with the bare metal provider being our implementation: https://github.com/talos-systems/sidero. Full disclosure, I am the CTO of Talos Systems.