AutoSpotting

we scale up to about 100 machines. We use spot instances EXTENSIVELY. And that configuration was tricky actually. Its been a couple of months now. Works pretty ok. k3s is actually pretty simple to use now. the tricky part was to integrate with https://github.com/kubernetes/cloud-provider-aws and https://github.com/DirectXMan12/k8s-prometheus-adapter The hardest part is to get it to work with spot instances. we use https://github.com/AutoSpotting/AutoSpotting to integrate with it.

There's an excellent implementation using AWS lambda to manage spot instances at https://github.com/AutoSpotting/AutoSpotting What's fantastic about the autospotting implementation is: 1. It dynamically replaces existing instances with spot instances just by setting a tag 2. Rather than replacing the instance with a fixed spot instance type, it will choose the cheapest that fits the requirements. 3. If there are no spot instances available that fit the requirements, it will spin up on demand instances until spot instances are available. If you have experience working with spot then you will know that these are really outstanding features that hopefully amazon will bake-in in the future.

Have a look at https://github.com/AutoSpotting/AutoSpotting or the FAQ section on https://autospotting.org If those don't answer your questions feel free to reach out to me and I'll do my best to explain further.

Shout-out for "AutoSpotting", which transparently re-launches a regular On-Demand ASG as spot instances, and will fall back to regular instances: https://github.com/AutoSpotting/AutoSpotting/ Combined with the fact that you can have an ASG with multiple instance types: https://aws.amazon.com/blogs/aws/new-ec2-auto-scaling-groups... Means that you can be reasonably certain you'll never run out of capacity unless AWS runs out of every single instance type you have requested, terminates your Spot instances, and you can't launch any more On-Demand ones. (and even so, set a minimum percentage of On-Demand in AutoSpotting to ensure you maintain at least some capacity)