cluster-api-provider-nested

Hard multi tenancy can't realistically be achieved in the same logical K8s cluster. And it is a moving target, which makes trying to secure it by admission controllers... not a great plan. One needs to look into things like VirtualClusters to even begin to consider hard multi-tenancy with potential hostile tenants(https://github.com/kubernetes-sigs/cluster-api-provider-nest...). That is just about the control plane. It doesn't even touch the data plane. How secure that is even with the extra layer, I do not know. Even in the VM land we have seen crazy VM escape exploits over the years.T

> K8S done right is literally designed for multi tenancy No it is not. Not in the way they are using it. There are two main use-cases. One is multiple teams, in which case they are bound by their company's policies and guardrails The second is multiple customers. But that also assumes they have no direct access to the cluster. A vendor would, instead, deploy multiple instances of a workload; the customers would not. Straight from the horse's mouth: https://kubernetes.io/docs/concepts/security/multi-tenancy/ There's also nothing that says that multiple clusters need to be expensive, if they are sized right. They can be as small as you need both in number of instances and instance size. The overhead here is the control plane but, for small clusters, the resource requirements are similarly small. That said, if hard multi tenancy is what you

Kubernetes has a lot of limitations from a multi tenancy perspective. It's functional, but I think it's not as polished as the rest of Kubernetes which is why Kubernetes has a multi tenancy SIG that spawned the hierarchical namespace controller (https://github.com/kubernetes-sigs/hierarchical-namespaces) and virtual clusters (https://github.com/kubernetes-sigs/cluster-api-provider-nest...)

You joke, but have you looked at K8s VirtualCluster? Separate control planes for each tenant within a K8s cluster. https://github.com/kubernetes-sigs/cluster-api-provider-nest...