cachey

er this limited our scalability in terms of concurrent readers and throughput, as well as implied cross-zone network costs when the zones of the gateway and stream-owning process did not align. The source of durability was S3, so there was a path to slurping recently-written data straight from there (older data would already be read directly), and take advantage of free bandwidth. But even S3 has RPS limits [6], and avoiding the latency overhead as much as possible is desirable. Caching helps reduce S3 operation costs, improves the latency profile, and lifts the scalability ceiling. Now, regardless of whether records are recent or old, our reads always flow through Cachey. Cachey internals: - It borrows an idea from OS page caches by mapping every request into a page-aligned range read. This did call for requiring the typically-optional Range header, with an exact byte range. Standard tradeoffs around picking page sizes apply, and we went with fixing it at the high end of S3's recommendation (16 MB). If multiple pages are accessed, some limited intra-request concurrency is used. The sliced data is sent as a streaming response. - It will coalesce concurrent requests to the same page (another thing an OS page cache will do). This was easy since foyer provides a native 'fetch' API that takes a key and thunk. - It mitigates the high tail latency of object storage by maintaining latency statistics and making a duplicate request when a configurable quantile is exceeded, picking whichever response becomes available first. Jeff Dean discussed this technique in "The Tail at Scale" [7], and S3 docs also suggest such an approach. A more niche thing Cachey lets you do is specify more than 1 bucket an object may live on, and attempt up to 2, prioritizing the client's preference blended with its own knowledge of recent operational stats. This is actually something we rely on since we offer regional durability with low latency by ensuring a quorum of zonal S3 express buckets for recently-written data, so the desired range may not exist on an arbitrary one. This capability may end up making sense to reuse for multi-region durability in future, too. I'd love to hear your feedback and suggestions! Hopefully other projects will also find Cachey to be a useful part of their stack. [1] https://github.com/foyer-rs/foyer [2] https://clickhouse.com/blog/building-a-distributed-cache-for... [3] https://turbopuffer.com/docs/architecture [4] https://www.warpstream.com/blog/minimizing-s3-api-costs-with... [5] https://risingwave.com/blog/risingwave-elastic-disk-cache [6] https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi... [7] https://cacm.acm.org/research/the-tail-at-scale/#body-7