sysbox

ecommend self hosting anything you do outside of your normal VPC. Sandboxes are the biggest possible attack surface and it is a feature of us that we're not in your cloud; If we mess up security your app is still fine.

VMs (the trade-off is performance). "Sysbox isolates the container in ways that gVisor does not" just isn't true. The sysbox approach is one kernel bug away from host system compromise, same as using regular containers. Emulating procfs and sysfs and using user namespaces takes away some of the attack surface and is great defense in depth, but does not provide isolation from the host kernel. > Also, note that Sysbox is not meant to isolate workloads in multi-tenant environments (for that we think VM-based approaches are better) I've read numerous claims that sysbox is suitable for untrusted workloads, for instance in [1] and [2]. It's a nice product and certainly much, much better than running docker-in-docker using privileged containers, but given the significant remaining attack surface, this claim could put your customers at risk and should come with a big disclaimer. > While QEMU may be faster than Firecracker in some metrics in a one-instance comparison, when you start running dozens of instances per host, the overhead of the full VM (particularly memory overhead) hurts its performance (which is the reason Firecracker was designed) Firecracker was designed for memory efficiency, faster cold start times and security (by virtue of being written in a memory-safe language). It means you can run more containers per host, but the actual workload performance overhead is identical to "normal" VMs and, in some cases, even slightly higher since Firecracker lacks some of the optimization that has gone into QEMU. > Regarding gVisor performance, we didn't do a full performance comparison vs. KubeVirt, so we may stand corrected if gVisor is in fact slower than KubeVirt when running multiple instances on the same host (would appreciate any more info you may have on such a comparison, we could not find one). KubeVirt is just plain QEMU VMs using libvirt, which have been compared to gVisor quite extensively[3][4]. There's almost no overhead for memory/CPU and quite a lot of overhead for syscalls (but with big improvements recently with the introduction of VFS2 and soon LisaFS[5]). It's a classic trade-off - gVisor is more secure and efficient than QEMU, allowing a much larger number of instances to run on a host by virtue of better cooperation with the host kernel scheduler and memory management, but for raw performance, a QEMU VM always wins. > Regarding the claim that standard containers cannot run a full OS, what the table in the GH repo is indicating is that Sysbox allows you to create unprivileged containers (or pods) that can run system software such as Docker, Kubernetes, k3s, etc. with good isolation and seamlessly (no privileged container, no changes in the software inside the container, and no tricky container entrypoints). To the best of our knowledge, it's not possible to run say Kubernetes inside a regular container unless it's a privileged container with a custom entrypoint. Or inside a Firecracker VM. If you know otherwise, please let us know. Firecracker runs a full Linux kernel inside the VM, so it could always run regular Docker, Kubernetes or anything else. See [6] for a practical example. For containers, this used to be the case, but the situation improved in recent kernel releases. For podman, almost every combination works - running systemd unprivileged, running podman inside podman, or even running rootless-podman-in-rootless-podman[7] and so does Kubernetes-in-rootless-{podman,docker}[8] (requiring very recent kernel features, though - notably cgroupsv2 and unprivileged overlayfs). Running docker:dind-rootless inside unprivileged Docker containers also works, however, it requires "--security-opt seccomp=unconfined". Sysbox definitely got to that point earlier and has better usability. > - Regarding "The claim that their solution offers large security improvements over any other solution with user namespaces isn't true". Where do you see that claim? The table explicitly states that there are solutions that provide stronger isolation. Apologies, then, for misinterpreting that. [1]: https://blog.nestybox.com/2020/10/06/related-tech-comparison... [2]: https://github.com/nestybox/sysbox/issues/120#issuecomment-9... [3]: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619... [4]: https://www.scitepress.org/Papers/2021/104405/104405.pdf [5]: https://gvisor.dev/blog/2021/12/02/running-gvisor-in-product... [6]: https://github.com/innobead/kubefire [7]: https://www.redhat.com/sysadmin/podman-inside-container [8]: https://kind.sigs.k8s.io/docs/user/rootless

, Docker, K8s, K3s, and more. This enables containers to replace slower/less-efficient VMs in many scenarios. Prior to Sysbox this required insecure privileged containers, custom images, and special host mounts, or specialized tools like LXD, KinD and Minikube. With Sysbox, the container runtime sets up the container such that it can run the software securely and seamlessly, increasing security and reducing complexity. Up to recently Sysbox only worked under Docker, but the latest release (v0.4.0) now works under Kubernetes too. This means you can use Kubernetes to orchestrate pods that are rootless (i.e., root in the container maps to an unprivileged user on the host) and can run not just microservices, but full "VM-like" environments. For example, you can create a pod that acts as a well isolated dev environment and inside of it run systemd, your favorite editor, plus Docker. Or create several pods that together form another K8s cluster for testing. Or run the K8s.io KinD inside a pod to create an entire K8s cluster inside one pod. Many interesting and powerful combinations are possible. Sysbox has taken 2-years of very hard work, as it pushes the limits of OS virtualization (uid-shifting, syscall trapping, procfs virtualization, etc.) It was forked from the OCI runc in 2019, so we stand on the shoulders of the developers of that excellent project. Would love to hear your feedback, if you think this new feature is useful, and for which use-cases. Would also encourage you to try it, we think you'll find it useful. Thanks! - Cesar & Rodny Sysbox: https://github.com/nestybox/sysbox Nestybox: https://www.nestybox.com/

y present in Ubuntu at the moment, but as i said, we can live without it. Which leads me to your question: why would you wait if the functionality you're after is already there? If having dockerd running as an unprivileged user is not a real must-have for you, then Sysbox provides a fairly secure solution while giving you all the functionality. Sorry, i'm not familiarized with vpnkit yet, will take a look. Correct, those are some of the benefits Sysbox-EE offers at the moment. That, plus efficiency & scalability features and hardened testing. Thanks a lot for your detailed feedback @geofft. Please ping us on slack anytime.