YCSB

Of course mileage may vary with different workloads, but are there any good benchmarks/suites to use for comparison in cases like these? They used YCSB but I don't know if those workloads ([1]) are relevant to modern/typical access patterns nor if they're applicable to SQL databases. You thinking about running some benchmarks in a bcachefs branch (:pray:)? I want to see this data structure prototyped in PostgreSQL. [1]: https://github.com/brianfrankcooper/YCSB/tree/master/workloa...

I used 100 as the scale factor and used different weights for benchmarks A through F, defined by the original YCSB project here: https://github.com/brianfrankcooper/YCSB/tree/master/workloa...

It's a broad question - but are there any industry standard database benchmark that companies use ? I have seen YCSB [https://github.com/brianfrankcooper/YCSB] - but don't know if there are others.

try YCSB benchmark : https://github.com/brianfrankcooper/YCSB dfsio: https://support.gopivotal.com/hc/en-us/articles/200864057-Ru... note: EBS IO performance used to be abysmal, especially for cheaper instances, use storage which comes native with the EC2 instances vs network-based, see AWS FAQ: http://www.datadoghq.com/wp-content/uploads/2013/07/top_5_aw... if you're interested in scalable in-memory computing (online vs batch) try Storm: http://storm.incubator.apache.org/ Spark: http://spark.apache.org/ Phoenix: http://phoenix.incubator.apache.org/

Comparing Hyperdex against Cassandra and MongoDB is appropriate. All three are distributed databases with secondary indexes and an emphasis on consistency and scalability. The other two databases you mentioned are not in the same problem domain. Redis will not manage more data than it can address in memory (barring the deprecated virtual memory paging) and Memcached doesn't bother with persistence at all. Both of these are very much memory databases targeting ephemeral storage tasks. Neither of them have distributed capabilities beyond very minimal master/slave configurations. The YCSB benchmarks [0] were developed for Yahoo to evaluate a wide range of NoSQL databases by abstracting a few common operations to measure the primitives shared by all of them. [0]: https://github.com/brianfrankcooper/YCSB

All requests are flushed to the server before the test is finished. So yes it does hit the database and average latency will be correct. https://github.com/brianfrankcooper/YCSB/blob/master/hbase/s... cleanup calls flushCommits. So all requests are there. No get requests are served from the client's buffers so all of those must hit the region server. In addition the client side write buffer size is 12mb. So we're not talking about all requests being buffered at a time. This may or may not be what is needed for your application so autoFlush is a per-client setting (HBase actually defaults it to on). Deferred log flushing is similar, though it's a per table setting.

You may want to check out the Yahoo cloud serving benchmark, it is a pretty standard load-testing tool for this kind of thing. https://github.com/brianfrankcooper/YCSB/wiki