s3backer

s or "objects". Most applications don't need the ability to access one or more bytes (in... a block) of an area on disk, they often only want access to entire files (roughly a 1:1 mapping with objects, with how 99% of the people use object storage). Getting back to the question, block storage is usually interacted with via a completely different set of technologies -- iSCSI[0][1], NVMeOF[2], etc. The idea here is that you're talking to a hard drive on the other end, and it make sense to speak the language (or a language) of hard drives. Object storage is normally interacted with via HTTP. The expected/tolerated latencies using these different technologies are different orders of magnitude. To rehash, object storage is similar, but seeks to up the level of abstraction and give you access to an consistent but opaque access to files. How is the object storage storing your files? You don't know and you don't care (probably) -- what you care about is how fast your HTTP request returns (or doesn't). This interface here is similar enough to reading and writing files locally but more importantly enables multiple applications to share the same storage without sharing the same hard drive. You can also write certain slices to files as well (this requires some more complicated locking and what not just like it would do locally). I want to also note that there's a conceptual step between "traditional" block storage and "new age" object storage that others might skip over, that's distributed storage systems like NFS. It's NFS's job to present a file system that when changed, prompts a file system on a completely different machine to change in the same fashion. It's easy to imagine a simple way to make functionality work but of course reality is more complicated. Object Storage arises from the realization that you don't actually have to interact with a "fake"/shimmed version of the local filesystem that appears local but is actually remote -- you can just send a HTTP request over the internet to a machine asking for the file you want when you want it. Here's a fun thing to think about -- is a database an implementation of object storage? You normally don't ship bytes to databases, you ship records (which you happen to decide the structure of) -- records are closer to files conceptually than they are to a "block" of bytes, even though at the end of the day the digital storage we're referring to is going to be bytes on a storage medium somewhere. If you really want to get a good instinct for this, dedicate some time to skimming/reading through the Ceph documentation and you'll see how they layer the object storage (and other things) on top of a lower level "block" management[3]. The picture on the first page should be quite instructive. [0]: https://www.truenas.com/docs/core/sharing/iscsi/ [1]: https://stonefly.com/blog/what-is-internet-small-computer-sy... [2]: https://blog.westerndigital.com/nvme-of-explained/ [3]: https://docs.ceph.com/en/latest/architecture/

ion issues with large datasets [3] Does anyone know of anything that can provide the above requirements? At this point it's really more of a thought experiment, as I've decided to go with Tarsnap [4] for my use-case. [0]: http://www.rath.org/s3ql-docs/impl_details.html [1]: https://defuse.ca/audits/encfs.htm [2]: https://github.com/archiecobbs/s3backer [3]: http://librelist.com/browser/attic/2015/3/31/comparison-of-a... [4]: https://www.tarsnap.com/