jdupes

200 lines of Nim [1] seems to run about 9X faster than the 8000 lines of C in fdupes on a little test dir I have. If you need C, I think jdupes [2] is faster as @TacticalCoder points out a couple of times here. In my testing, `dups` is usually faster than `jdupes`, though (but usually only ~1.5..2.0X faster in hot cache settings -- for this problem "really slow" usually comes from IO, though). [1] https://github.com/c-blake/bu/blob/main/dups.nim [2] https://github.com/jbruchon/jdupes

"This repository has been archived by the owner on Oct 6, 2023. It is now read-only." According to https://github.com/jbruchon/jdupes/commit/2984bd77f4712359, it's at https://codeberg.org/jbruchon/jdupes now.

I've also seen https://github.com/jbruchon/jdupes used

My research into this many years ago turned out that jdupes was the right / best solution I could find for my usecase. https://github.com/jbruchon/jdupes Though that works fine from a script perspective I'd like some more interactive way of sorting directories etc. Identifying is just the first step, jdupes helps with linking the files (both soft and hard links comes with caveats though!) but that is mostly to save space, not to help in reorganisation.

I like fdupes/jdupes for finding duplicate files: https://github.com/adrianlopezroche/fdupes https://github.com/jbruchon/jdupes

I use and test assorted duplicate finders regularly. fdupes is the classic (going way way back) but it's really very slow, not worth using anymore. The four I know are worth trying these days (depending on data set, hardware, file arrangement and other factors, any one of these might be fastest for a specific use case) are: https://github.com/jbruchon/jdupes https://github.com/pauldreik/rdfind https://github.com/jvirkki/dupd https://github.com/sahib/rmlint Had not encountered fclones before, will give it a try.

I've used it for backups for five years now. My USB backup drives have gotten full quite a lot of times because of old snapshots (rsync isn't able to detect moving directories), and I've had zero problems with it – just have to delete some old snapshots and wait a bit and then there's space again. I don't use RAID5, which is where I've heard btrfs is dangerous, just for the snapshots and dedupe (with https://github.com/jbruchon/jdupes/ )

fdupes -rL /dir see also jdupes fork of fdupes https://github.com/jbruchon/jdupes https://github.com/adrianlopezroche/fdupes