goreplay

in the past, when a single Redis instance would have done it. Specially if it's temporary, non-critical data such as spam detection. * Do you batch the incoming events before triggering SQS? * How are your networking costs going? They tend to creep up in the AWS bills too. It just sounds to me you clearly need something in-between the public endpoint and those Lambdas. But maybe I'm missing some info. Otherwise you're going to keep paying for capacity that a "traditional" server could handle, without costing you more for each request. If you can't fight back, and don't want real requests to be lost, put a server at the battle-front, and buffer all raw HTTP requests into a queue made for this (like Kinesis). You can then process events and recover the data at your own pace. I did this in the past, and could handle 60k+ req/s with 2 instances on AWS. I used a simple Go tool to capture the traffic https://github.com/buger/goreplay Also, at my full-time job we use Kinesis to buffer all analytics events, it's cheaper than SQS and handles billions of data points per month. This also keeps the ingest rate constant, but you seem to already do that with your workers. Full disclosure: I'm Anthony from http://panelbear.com/ , and I just want to offer honest help.

tps://docs.mitmproxy.org/stable/ There is a whole class of "VCR" projects for recording traffic, these tend to be language specific (VCR is in Ruby), but there are ports to other languages: https://github.com/vcr/vcr https://github.com/bblimke/webmock The vendor products tend to be labelled Service Virtualization. I used to work for one of those companies, ITKO, we were acquired by CA Technologies (now Broadcom) in 2011. There are vendor products from Micro Focus, Tricentis, Broadcom, Parasoft, etc. It's important to think about your use case: local development, unit testing, CI, integration testing, performance testing, recording vs. programming, protocol support, payload support, etc. Many of the tools focus on just a subset of these areas.

of `fluentd` and put stuff like user id, device id into tag... I want to log slow query time, now, I cannot use whole query as tag because of very high cardinality. However, I kept using InfluxDB. I want to support it so we have something better than Postgres. I'm sick of SQL query(personaly) and I would like for Flux to be successful. Similar to MongoDB, it's bad years ago but very good nowadays. I guess same thing will happen with InfluxDB. And indeed, they do improve over the year. Similar to how we use Ruby vs C. It's about productivity. And despite of above pain points, I always find a way to solve them eventually. And the tooling around InfluxDB is nice, especially Grafana.