aifiles

iple tar.gz archives, zip archives, etc, which (after extraction/unencryption) will share 20% files here, 10% files there, a 4kb jpg that's the same as a 100MB PNG here and there, etc. So yes, those 10TB archives may end up being 5TB if someone spent the time to really comb over, understand, make good decisions, and organize that data. But I have not yet seen anything that can scratch that surface yet, other than perhaps https://github.com/jjuliano/aifiles - but I won't use it until it's local only and has guarantees not to destroy data without explicit permission. An overlay filesystem that shows compression/deduplication with LLM capability like aifiles is probably the best option here. However, I wouldn't imagine that most people's life data is less than 2TB even with all of this - it's mostly imposed as an artificial constraint by these companies.

uplicates' to be removed, and how to better compress data As for the de-duplication and processing: this could be very good for dropbox in that, e.g. if a person wants to completely re-encode all of their image files or video files with AV1, the resulting data could be cut in half or more - which saves Dropbox storage space. After which, neural perceptual hashing could be done on all of the files and a threshold of similarity could do de-duplication on a perceptual basis (for example, keep the bigger size file that is 99% similar to a 2x downsized version, and re-encode it). User preference to keep things like tiff files completely intact or any other lossless encoding of their choosing could be good options as well There's definitely a strange disparity between the computation cost for deploying a decent model to do this compared to the storage cost - but if a (perhaps even non-LLM!) small model is created to be able to plow through data at fast rates could be deployed it may make sense. Or perhaps the type of semantic compression that LLMs do are of interest for making a new type of lossy compression algorithm of which Dropbox is interested. It is certainly interesting that even the largest LLMs we are aware of contain only a few hundred GBs of weights, but have information on various topics from Pokemon to electric engineering, to cellular biology, to cake decorating - yet one instance of the best compressed pubmed database I can think of (only containing the small fraction of cell bio and med part of the LLM - perhaps 2% of it) is like 60GB with Zstd compressed (yes, yes, I know lossy vs lossless - but still) - which would be probably more like 1/2 or 1/4 of a typical SoTA LLM. So clearly lossy compression via including semantic info is hugely beneficial over typical lossy compression techniques (and perhaps one could argue, what often lossy compression algorithms try to model in the first place, in a very rudimentary way).