markfasheh

duperemove

C2015–2025

markfasheh / duperemove

duperemove

Tools for deduping file systems

Systems & Infrastructure

Explore on GitHub Read Hacker News discussions

9Distinct people

9Mentions

0Explicit recommenders

1KGitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Tools for deduping file systems

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year9 dated mentions

Observed mentions: 2015-05-24 09:41:51 UTC to 2025-02-25 21:28:52 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
2015	2015-01-01 00:00:00 UTC	2016-01-01 00:00:00 UTC	1	0	1
2016	2016-01-01 00:00:00 UTC	2017-01-01 00:00:00 UTC	0	0	0
2017	2017-01-01 00:00:00 UTC	2018-01-01 00:00:00 UTC	0	0	0
2018	2018-01-01 00:00:00 UTC	2019-01-01 00:00:00 UTC	1	0	1
2019	2019-01-01 00:00:00 UTC	2020-01-01 00:00:00 UTC	0	0	0
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	0	0	0
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	0	0	0
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	0	0	0
2023	2023-01-01 00:00:00 UTC	2024-01-01 00:00:00 UTC	4	0	4
2024	2024-01-01 00:00:00 UTC	2025-01-01 00:00:00 UTC	2	1	1
2025	2025-01-01 00:00:00 UTC	2026-01-01 00:00:00 UTC	1	0	1

What Hacker News says

9 sources

kevincox·Feb 25, 2025

Yes, Linux has a systemcall to do this for any filesystem with reflink support (and it is safe and atomic). You need a "driver" program to identify duplicates but there are a handful out there. I've used https://github.com/markfasheh/duperemove and was very pleased with how it worked.

In Hyperspace

nikisweeting·Oct 31, 2024

- https://github.com/markfasheh/duperemove - https://codeberg.org/jbruchon/jdupes / https://www.jdupes.com/ - https://github.com/adrianlopezroche/fdupes - https://github.com/pauldreik/rdfind

In OpenZFS deduplication is good now and you shouldn't use it

anotherhue·Aug 8, 2024Submission

Duperemove – Tools for deduping file systems

54 points · Open discussion

jeroenhd·Nov 5, 2023

Very useful for identifying files that may need to get deduplicate or that can be removed entirely. Unfortunately, I don't think this will also find identical directories. If deleting files isn't what you want, I'd suggest looking into deduplicating tools. ZFS has its own de duplicator built in, which is nice. It should just deduplicate files and individual extents of files by itself once you enable it. Probably not a good idea on very write-heavy disks, but it's an option. Other file systems with extent level deduplication can use https://github.com/markfasheh/duperemove to not only deduplicaye files, but also deduplicate individual extents. This can be very useful for file systems that store a lot of duplicate content, like different WINE prefixes. For filesystems without extent deduplication, duperemove should try hard linking files to

Read the rest

make them take up practically no disks space.

In Fdupes: Identify or delete duplicate files

ptman·Oct 31, 2023

ZFS now has reflink support, which doesn't require lots of RAM, but isn't done automatically while writing. You need to run something like https://github.com/markfasheh/duperemove

In Bcachefs Merged into the Linux 6.7 Kernel

the8472·Jul 4, 2023

Any copy command. On-demand deduplication managed by userspace. https://man7.org/linux/man-pages/man2/ioctl_fideduperange.2.... https://man7.org/linux/man-pages/man2/copy_file_range.2.html https://github.com/markfasheh/duperemove

In ZFS 2.2.0 (RC): Block Cloning merged

JackSlateur·Mar 26, 2023

Looks like https://github.com/markfasheh/duperemove

In Craziest thing I ever used SQLite for: partial file deduplication (2022)

newnewpdro·May 29, 2018

With XFS and reflink, out-of-band deduplication is totally possible and is a userspace [1] issue. But XFS is not doing anything to assist in accelerating the identification of duplicate blocks, instead it simply implements ioctls for what is essentially extent sharing. [1] https://github.com/markfasheh/duperemove

In About ZFS Performance

simoncion·May 24, 2015

So, I'm asking questions because I'm curious and very probably ignorant. I'm not trying to pick a fight, make points, or get in a dick-waving contest. I also don't know how closely you follow btrfs development. I skim the lists from time to time, so if I'm telling you stuff that you already know, or if my memory is not quite correct, I apologize in advance. Also, if you notice this comment after the comment submission window closes, you can reach me at $MY_HN_USERNAME@gmail What's wrong with how compression works? From a user's perspective, you either set a mount option, set a bit on a file attribute, or explicitly call for compression with btrfs defrag. [0] If you combine the latter two operations into a single checkbox, this is exactly how NTFS handles things. What am I missing here? Also, I can't agree that modern[1] btrfs handles unex

Read the rest

pected power cuts poorly. This just hasn't been my experience. I can, however, agree that log replay isn't complete and still needs work. I can't speak to the ENOSPC bugs, I haven't run into any in a very, very long time. Some time after 3.14, btrfs got a pool of space called the "Global Reserve" [2] which was intended to address ENOSPC issues. Somewhere around that time, btrfs also grew a better btrfs-specific df function invoked with "btrfs filesystem usage". I don't make extensive use of snapshots, but the numbers I get out of btrfs fi usage almost exactly match the numbers I get from plain old df. I'm unware of snapshot automounting. Can you help me understand what this is? (A brief Google search wasn't enlightening.) I agree that taking away the ability to have subvolumes outside the FS tree if you didn't set things up just right at FS creation time is pretty bullshit. That certainly needs reworked. Everything else in my limited experience with subvol management seems okay to me. What's seems strange to you? Btrfs doesn't yet have built-in online dedup, but are you aware of dupremove? [3] Can you give me a recipe to create an N(N-1)/2 sized random access file? I use BTRFS for some very small random-access databases and large pretty-much-append-only databases and haven't run into this behavior. I have a couple of things to say about your other comment: People generally don't write to a mailing list to say what a good time they're having with your software. It's pretty shit that log replay isn't worked out yet. I certainly* don't intend to stop making backups. OTOH, I've been "getting lucky" for five years straight while running a FS that many people regard as the most untrustworthy thing in the world on a drive that many people regard as entirely unreliable and untrustworthy. :) Cheers! [0] Conceptually, this makes a lot of sense, 'cause all defrag does is re-write the file. NTFS re-writes files when you request that they be compressed, too. [1] "Modern" means btrfs from the past two years or so. [2] Much like ext*'s reserved-for-the-superuser space. [3] https://github.com/markfasheh/duperemove

In Ext4: fix data corruption caused by unwritten and delayed extents