rmlint

> https://github.com/sahib/rmlint is the one I had in mind. Ah, right, thanks. I now dimly recall some old project realizing fs-snapshots using hard links, which one could consider some sort of deduplication as well. > I think you are thinking of block level online deduplicators that are integrated into the file system? Indeed, I was.

https://github.com/sahib/rmlint is the one I had in mind. > Those use a rather expensive hash function (you really want to avoid hash collisions), [...] Then we are clearly not thinking of the same kind of software. > but (at least some ten years ago) memory, not processing speed, was the limiting factor. In what I described, IO is the limiting factor. You want to avoid having to read the whole file, if you can. I think you are thinking of block level online deduplicators that are integrated into the file system?

See the comments on https://news.ycombinator.com/item?id=38113396 for a list of alternatives. I used https://github.com/sahib/rmlint in the past and can't complain.

It does not support APFS: https://github.com/sahib/rmlint/issues/421

I'm pretty sure some of them also work on MacOS. rmlint[1], for example can output a script that reflinks duplicates (or run any script for both files): rmlint -c sh:handler=reflink . I'm not sure if reflink works out of the box, but you can write your own alternative script that just links both files [1]: https://github.com/sahib/rmlint

They can be, pip developers just have to care about this. Nothing you described precludes file-based deduplication, which is what pnpm does for JS projects: it stores all library files in a global content-addressable directory, and creates {sym,hard,ref}links depending on what your filesystem supports. Being able to mutate files requires reflinks, but they're supported by e.g. XFS, you don't have to go to COW filesystems that have their own disadvantages. You can do something like that manually for virtualenvs too, by running rmlint on all of them in one go. It will pick an "original" for each duplicate file, and will replace duplicates with reflinks to it (or other types of links if needed). The downside is obvious: it has to be repeated as files change, but I've saved a lot of space this way. Or just use a filesystem that supports dedup

My preferred solution is rmlint [https://github.com/sahib/rmlint] mostly because it also looks at duplicate directories. It produces a bash script instead of deleting anything itself, so you can examine it before running the script it made.

ZFS is COW with checksums so it wouldn't edit the same blocks and there is the possibility for sending snapshots to another pool (which may/may not use deduplication). Although, deduplication comes with a performance cost. I had all my photos spread out on various disks and external media, sometimes extra copies (as I did not trust certain disks) - if I remember it correctly I went from 3.6T to 2.2T usage by consolidating all my photos to a deduplicated pool. All fine, but the zpool wanted way more RAM and felt slower than my other pool. After I removed duplicates (with help of https://github.com/sahib/rmlint ), I migrated my photos to an ordinary zpool instead.

rmlint. I've tried many, but rmlint is the most flexible and reliable. Esp. the tagging works really well. https://github.com/sahib/rmlint

I use and test assorted duplicate finders regularly. fdupes is the classic (going way way back) but it's really very slow, not worth using anymore. The four I know are worth trying these days (depending on data set, hardware, file arrangement and other factors, any one of these might be fastest for a specific use case) are: https://github.com/jbruchon/jdupes https://github.com/pauldreik/rdfind https://github.com/jvirkki/dupd https://github.com/sahib/rmlint Had not encountered fclones before, will give it a try.

I've found rmlint to be another very good tool in this space: https://github.com/sahib/rmlint

rmlint[1] works extremely well for duplicated files (including photos) [1] https://github.com/sahib/rmlint

Tools like rdfind and rmlint breathed new life into my SSD by hardlinking duplicate files after video editing software helpfully duplicated huge videos on import. https://rdfind.pauldreik.se/ https://github.com/sahib/rmlint There are great GUI visualizers for disk space, like WizTree/Baobab; for the command line I've had success with ncdu. https://dev.yorhel.nl/ncdu