wrk2

10 years old and still relevant. Gil created a wrk fork https://github.com/giltene/wrk2 to handle coordinated omission better. I used using his fork for many years. But I think he stopped updating it after a while. Good load testing tools will have modes to send in data at a fixed rate regardless of other requests to handle coordinated omission. k6 for instance defined these modes are "open" and "closed": https://grafana.com/docs/k6/latest/using-k6/scenarios/concep.... They mention the term "coordinated omission" on the page however I feel like they could have given a nod to Gil for the inventing term.

Sounds like macOS sucks at web serving :) I guess one lesson not explicitly mentioned is that doing benchmarking on something like macOS can have lot of pitfalls compared to systems more intended for this kind of workload (Linux...). But the mention of wrk immediately stood out as a red flag; there is notable problem with wrk called "coordinated omission": https://github.com/wg/wrk/issues/485 https://github.com/giltene/wrk2 https://news.ycombinator.com/item?id=34148502 Benchmarking is difficult. And there is lot of subtlety on how you set up the client(s) and servers that can impact the results a lot.

If you want a simple load testing tool for HTTP, use wrk2[1]. wrk -t2 -c100 -d30s -R2000 http://127.0.0.1:8080/index.html > This runs a benchmark for 30 seconds, using 2 threads, keeping 100 HTTP connections open, and a constant throughput of 2000 requests per second (total, across all connections combined). Some distros include `ab`[2] which is also good, but wrk2 improves on it (and on wrk version 1) in multiple ways, so that's what I use myself. [1] https://github.com/giltene/wrk2 [2] https://httpd.apache.org/docs/2.2/programs/ab.html

I've had my eyes on wrk2 [1] 1. https://github.com/giltene/wrk2 But I am curious, what does HN use? Any tips?

I agree, sounds like a good approach. Can the instance do 2 GB/s to disk at the same time it is doing 3.1GB/s across the network? Is that bidirectional capacity or on a single direction? How many threads does it take to achieve those numbers? That is kind of a nice property, that the network has 50% more bandwidth than the disk. 2x would be even nicer, but that turns out to be 1.5 and 3, so a slight reduction in disk throughput. Are you able to run a single RP Kafka node and blast data into it over loopback? That could isolate the network and see how much of the available disk bandwidth a single node is able to achieve over different payload configurations before moving on to a distributed disk+network test. If it can only hit 1GB/s on a single node, you know there is room to improve in the write path to disk. The other thing that people

Note that Gil Tene has his own wrk fork. The readme has some interesting notes about wrk: https://github.com/giltene/wrk2 There is also open issue on GH questioning wrks approach for resolving coordinated omission: https://github.com/wg/wrk/issues/485 This is a deep, ultimately unsatisfying, rabbit hole to fall into

Needs (2015). I loved the talks from Gil Tene. I always reach for his fork of wrk whenever I need to test throughput: https://github.com/giltene/wrk2

i use https://github.com/giltene/wrk2 pretty regularly. it has decent lua hooks to customize behavior but i use it in the dumbest way possible to hammer a server at a fixed rate with the same payload over and over. i run it by hand after a big change to the server to make sure nothing obviously regressed. i used to run it nightly in a jenkins job but 99% of the time no one looked at results. it was nice to see if assumptions on load a single node could handle didn't hold anymore.

I so deeply wish for a set of benchmarks that accepted only idiomatic, standard code that a slightly above average developer would plausibly write. Pretty much every set of benchmarks out there is ruined by monstrous unrealistic code that squeezes out 5% more performance than actually well written code. As a side note, one flaw of TechEmpower benchmarks is that to my knowledge they use the original "wrk" tool, which only supports HTTP 1.0 and is sensitive to "coordinated omission" [0]. This means it ends up being biased in favor of web frameworks that implement specific optimizations or have specific behaviors that aren't actually useful in a production context. [0] https://github.com/giltene/wrk2

I'm not disagreeing with your results, but you should be using https://github.com/giltene/wrk2 based benchmarks to avoid coordinated omission errors in measuring latency.

Did you consider wrk2? https://github.com/giltene/wrk2 Maybe you duplicated some of these fixes?

> I know there's much better ways of testing load/performance. such as https://github.com/giltene/wrk2