autoscaler

Would CAPH solve your issue? https://github.com/syself/cluster-api-provider-hetzner you would wire up cluster autoscaler to point to it using clusterapi https://github.com/kubernetes/autoscaler/tree/a9cb59fdd2dd88...

> * Hetzner: cheap, good service, the finest pets in the world, no cattle You can absolutely do cattle with Hetzner. They support imaging and immutable infrastructure. They don't have a native auto scaling equivalent, but if you're using Kubernetes, they have a cluster autoscaler: https://github.com/kubernetes/autoscaler/blob/master/cluster... Hetzner Cloud's biggest gap for me right now is secure VMs: they don't support encrypted disks, UEFI, secure boot, or TPMs (and certainly nothing like AMD SEV). You can get a fat bare metal Ryzen box through Robot and DIY though.

In particular an ephemeral directory mounted on e.g. /tmp counts as "local storage", so in common configurations nothing will get evicted until you work around it by adding "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" to all deployments. Silly default IMO but apparently "intentional". https://github.com/kubernetes/autoscaler/issues/2048

This might be true for "stock" Kubernetes, but it seems likely that the grandparent post is using cluster autoscaling (since they mention "auto-scaled up"), for which this post is not accurate. The cluster-autoscaler component does remove nodes with low utilization and Pods fitting onto other nodes, effectively defragging the cluster. There's just plenty of reasons for it to not drain a node (as per its FAQ[1]). But it's highly configurable, so I wonder if the grandparent post just needed some configuration changes to suit their scenario better. [1] https://github.com/kubernetes/autoscaler/blob/master/cluster...

GCE https://kubernetes.io/docs/concepts/cluster-administration/c... GKE https://cloud.google.com/container-engine/docs/cluster-autos... AWS https://github.com/kubernetes/autoscaler/blob/master/cluster... Azure https://github.com/kubernetes/autoscaler/blob/master/cluster... Alibaba Cloud https://github.com/kubernetes/autoscaler/blob/master/cluster... Brightbox https://github.com/kubernetes/autoscaler/blob/master/cluster... OpenStack Magnum https://github.com/kubernetes/autoscaler/blob/master/cluster... DigitalOcean https://github.com/kubernetes/autoscaler/blob/master/cluster... CloudStack https://github.com/kubernetes/autoscaler/blob/master/cluster... Exoscale https://github.com/kubernetes/autoscaler/blob/master/cluster... Equinix Metal https://github.com/kubernetes/autoscaler/blob/master/cluster... OVHcloud https://github.com/kubernetes/a

What are your comments on the following links below: - https://github.com/kubernetes/kubernetes/issues/51135 - https://github.com/libero/reviewer/issues/1023 - https://medium.com/@betz.mark/understanding-resource-limits-... - https://medium.com/omio-engineering/cpu-limits-and-aggressiv... This doesn't seem as dangerous as is being suggested -- and in a world with the kernel bug and some separation of workloads it seems very viable. Obviously, in a world without the kernel bug it makes much less sense to not set limits, but as far as scheduling goes, well-set requests (+/- a VPA[0]), with a HPA[1] should be enough to handle sudden increases in scale, and for truly large increases that are a complete surprise (otherwise you could have planned for it) elastic infrastructure via a cluster autoscaler[2]. [0]: https://github.com/kubernetes/auto

As an unsolicited recommendation, if you _wanted_ to attempt to chase this down, you should look at using the Vertical Pod Autoscaler (https://github.com/kubernetes/autoscaler/tree/master/vertica...) in recommendation only mode in order to get a good idea of what services are actually using, and set new limits based on that. It definitely won't be optimal, but should let you get to a place where you at least have some limits set.

If your pods need to be long running, you can annotate them as such and they will not be autoscaled. https://github.com/kubernetes/autoscaler/blob/master/cluster...

Check out Cluster AutoScaler: https://github.com/kubernetes/autoscaler/tree/master/cluster... This tool allows you to autoscale the cluster itself with various cloud providers. There is a list of cloud providers it supports at the end of the readme.

Efficiency is an area where there can be a lot of cost hiding. We recently saved a lot of money by: - Using the Kubernetes Vertical Pod Autoscaler (https://github.com/kubernetes/autoscaler/tree/master/vertica...) for CPU and memory scaling, and switching to metrics like requests per second and connection count for horizontal scaling - Collecting metrics on container CPU and memory allocation vs utilization - Writing scripts to auto-submit PRs to SREs with better recommended sizing based on actual usage - Tuning our VM instance sizes and autoscaler configs A few engineers were able to save the company several times their salary with a few months of work, and plan to 10x the savings over the next year

Yes, though the problems are real. In its current form Kubernetes can only handle a cluster size of 5,000 nodes: https://kubernetes.io/docs/setup/cluster-large/ and high-churn batch workloads don't work well. Kubernetes does support elastic resource sharing (assuming I understand what they need) via horizontal pod scaling and the cluster autoscaler: https://github.com/kubernetes/autoscaler/tree/master/cluster.... Although there are rough edges here, I was surprised by the quality of the code in the cluster autoscaler. I kept thinking of new edge cases and expected it to fall over, only to find out that they'd thought of that problem and it handled it gracefully. For batch workloads something like Argo can help: https://github.com/argoproj/argo. So basically I agree, it probably would've made more sense to build something on top of Kuberne

Autoscaler(https://github.com/kubernetes/autoscaler/tree/master/cluster...) is a good option too. Works incredibly well at automatically adding/removing nodes based on workload.

Running K8S on other environments isn't a difficult job, but you do need a certain level of experience running systems. You need to set up EtcD, in prod you have 3 or 5 of them - there are a couple of topologies for running EtcD and peer discovery (you will need to RTFM). This is all your K8S state, so you need to set up backups for these. Each node you are going to include in your cluster needs to have docker and kubelet running. You will need to start kubelet with your init system (systemd, etc). Then all you need is to bootstrap a control plane. The control plane has 3 components. The API server needs to communicate to etcd, everything else just talks to the API server. The API server is stateless and can (should) have multiple instances running. The other two parts are the Controller and Scheduler. Also stateless. The tool Bootkube[1]

Yes, this is my biggest problem with the autoscaler as well. I opened an issue about this almost a year ago, looks like they're finally getting around to addressing it: https://github.com/kubernetes/autoscaler/pull/77

It sounds like you may be looking for the cluster autoscaler: https://github.com/kubernetes/autoscaler/tree/master/cluster.... When used with Kops, it should give you what you're looking for (with the baked in kops AMI)