TItiagod·Oct 31, 2024
I run rdfind[1] as a cronjob to replace duplicates with hardlinks. Works fine!
https://github.com/pauldreik/rdfind
In OpenZFS deduplication is good now and you shouldn't use itpauldreik
rdfind
C++2021–2024
pauldreik / rdfind
rdfind
find duplicate files utility
6Distinct people
8Mentions
0Explicit recommenders
1.3KGitHub stars
What to know before exploring
What it is
find duplicate files utility
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year8 dated mentions
Observed mentions: 2021-08-30 20:12:42 UTC to 2024-10-31 00:48:57 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 3 | 1 | 2 |
What Hacker News says
8 sources- https://github.com/markfasheh/duperemove
- https://codeberg.org/jbruchon/jdupes / https://www.jdupes.com/
- https://github.com/adrianlopezroche/fdupes
- https://github.com/pauldreik/rdfind
In OpenZFS deduplication is good now and you shouldn't use itRdfind: A utilty to find duplicate files, delete them or replace with hardlinks
2 points · Open discussionA repurposed HP Pavillion TP01-0066 Desktop w/Ryzen 3700X, 256GB NVME SSD and 32GB RAM, with 5x USB 3.0 HDD's attached for mass storage and onsite backup using dirvish[1] and rdfind[2]. It has a fan but is usually silent.
[1] https://dirvish.org/
[2] https://github.com/pauldreik/rdfind
In Ask HN: What hardware are you running for your home server?I use rdfind to deal with this: https://github.com/pauldreik/rdfind
In Pip and cargo are not the sameI use and test assorted duplicate finders regularly.
fdupes is the classic (going way way back) but it's really very slow, not worth using anymore.
The four I know are worth trying these days (depending on data set, hardware, file arrangement and other factors, any one of these might be fastest for a specific use case) are:
https://github.com/jbruchon/jdupes
https://github.com/pauldreik/rdfind
https://github.com/jvirkki/dupd
https://github.com/sahib/rmlint
Had not encountered fclones before, will give it a try.
In Go Find Duplicates: A fast and simple tool to find duplicate filesAs far as I know, the standard tool for this is rdfind. This new tool claims to be "blazingly fast", so it should provide something to show it. Ideally a comparison with rdfind, but even a basic benchmark would make it less dubious. https://github.com/pauldreik/rdfind
But the main problem is not the suspicious performance, it's the lack of explanation. The tool is supposed to "find duplicate files (photos, videos, music, documents)". Does it mean it is restricted to some file types? Does it find identical photos with different metadata to be duplicates? Compare this with rdfind which clearly describes what it does, provides a summary of its algorithm, and even mentions alternatives.
Overall, it may be a fine toy/hobby project (3 commits only, 3 months ago), I didn't read the code (except for finding the command-line options). I don't get
Read the rest
why it got so much attention.