cloud-provider-aws

There are few paths we can take here. The CCM's primary responsibility is to implement the cloudprovider.Interface[0] which has node, route, and service controllers. However, the CCM can run arbitrary named controllers as well, giving us a fun extension point for the future. The in-tree CCMs are being phased out in favor of out-of-tree CCMs. The AWS Cloud Provider[1] is a good example of what that's starting to look like. My colleague demo'd Karpeneter internally. We haven't committed releasing it yet but we're discussing it. [0] https://github.com/kubernetes/cloud-provider/blob/master/clo... [1] https://github.com/kubernetes/cloud-provider-aws

we scale up to about 100 machines. We use spot instances EXTENSIVELY. And that configuration was tricky actually. Its been a couple of months now. Works pretty ok. k3s is actually pretty simple to use now. the tricky part was to integrate with https://github.com/kubernetes/cloud-provider-aws and https://github.com/DirectXMan12/k8s-prometheus-adapter The hardest part is to get it to work with spot instances. we use https://github.com/AutoSpotting/AutoSpotting to integrate with it.