ambry

There is also Ambry ( https://github.com/linkedin/ambry ) as an alternative. The blob store open-sourced, created and maintained by LinkedIn. It also has an S3 compatible interface. I think it is about 10 years old now and it is really stable.

Short answer: Yes. However, Ambry doesn't use byte-ranges for the upload. Each chunk can be uploaded separately and the client then requests a _stitch_ operation[1]. In that operation, the client specifies the list of blob IDs for each chunk (in order) and asks Ambry to create a metadata blob listing those IDs. Ambry then returns the blob ID for the metadata blob. [1]: https://github.com/linkedin/ambry/blob/9b7a49ac79b1678fd7fd7...

Internally, Ambry stores large objects as a series of chunks. At LinkedIn we've found that 4mb makes a good chunk size. So (for example), your 40mb upload could be streamed directly where Ambry would handle chunking it, or you could chunk it yourself. Client-side chunking has the advantage of cleaner resumption from a broken connection. After all the chunks are uploaded, Ambry creates (or the client asks Ambry to create[1]) a special metadata blob which contains a listing of all the chunk blob IDs. Clients can use the blob ID of the special metadata blob as a single reference to the large (composite) object without needing to worry about the underlying chunking for GET and DELETE operations. Depending on context, anything larger than the chunk size could be considered a large object, but using this method Ambry easily handles gigabyt

The readme at https://github.com/linkedin/ambry starts with: > Ambry is a distributed object store that supports storage of trillions of small immutable objects (50K -100K) as well as billions of large objects. 1. What is considered to be the size of a “large object”? 2. Can Ambry handle large object multipart uploads or would one have to build this themselves by storing chunks as separate objects?

I have worked on Ambry for a few years now. This paper is a bit old, but captures most of the core concepts of Ambry. Some of the most fascinating parts of the journey after this paper have been scaling the system to support hundreds of GiBps of throughput per cluster, multiple workloads supporting other databases and stream processing systems, and rethinking the replication to make it compatible with public cloud. Checkout the GitHub repo to learn more https://github.com/linkedin/ambry

We're using LinkedIn's Ambry ( https://github.com/linkedin/ambry ). Its an easy to use distributed object store. It's basically a self hosted version of Amazon S3.