concurrency-limits

I’d like to mention it since Uber has a really cool load shedder [1], also implemented similarly by Netflix [2] and failsafe-go [3]. It basically looks for points where more requests per second suddenly cause a significant increase in latency and calls that the concurrency limit. 1: https://www.uber.com/us/en/blog/cinnamon-using-century-old-t... 2: https://github.com/Netflix/concurrency-limits 3: https://failsafe-go.dev/adaptive-limiter/

TLDR: "Nginx will fail by refusing or dropping connections, Caddy will fail by slowing everything down" To me it seem that Caddy suffer from BufferBloat. Under heavy congestion the goodput (useful throughput) will drop to 0 because client will start timing-out before the server get a chance to respond. Caddy should use an algorithm similar to : https://github.com/Netflix/concurrency-limits Basically check what was the best request latency, and decrease concurrency limit until latency stop improving.

Perfect time to ask, I've been toying with using the netflix concurrency limits library for a while as opposed to the manual tuning of threadpools, queue depths, etc to achieve good utilization at certain latency. Curious if others have experience with it, and their thoughts: https://github.com/Netflix/concurrency-limits FWIW envoy also has an adaptive concurrency experimental plugin that seems similar that I'd also love to hear about any real world experience with: https://www.envoyproxy.io/docs/envoy/latest/configuration/ht...