dedup

Thank you for creating and sharing this utility. I ran it over my Postgres development directories that have almost identical files. It saved me about 1.7GB. The project doesn't have any license associated with it. If you don't mind, can you please license this project with a license of your choice. As a gesture of thanks, I have attempted to improve the installation step slightly and have created this pull request: https://github.com/ttkb-oss/dedup/pull/6

To make dedup[0] fast, I use a tree with device id, size, first byte, last byte, and finally SHA-256. Each of those is only used if there is a collision to avoid as many reads as possible. dedup doesn’t do a full file compare, because if you’ve found a file with the same size, first and last bytes, and SHA-256 you’ve also probably won the lottery several times over and can afford data recovery. This is the default for ZFS deduplication and git does something similar with size and far weaker SHA-1. I would add a test for SHA-256 collisions, but no one has seemed to find a working example yet. 0 - https://github.com/ttkb-oss/dedup

If you don’t mind CLI tools, You can try dedup - https://github.com/ttkb-oss/dedup . Use the —-dry-run option to get a list of files that would be merged without modifying anything and how much space would be saved.

I made a command line utility called `dedup` a while back to do the same thing. It has a dry-run mode, will “intelligently” choose the best clone source, understands hard links and other clones, preserves metadata, deals with HFS compressed files properly. It hasn’t destroyed any of my own data, but like any file system tool, use at your own risk. 0 - https://github.com/ttkb-oss/dedup

Did you consider the fts[0] family of functions for traversal? I use that along with a work queue for filtered entries to get pretty good performance with dedup[1]. For my use case I could avoid any separate stat call altogether, the FTSENT already provided everything I needed. 0 - https://linux.die.net/man/3/fts_read 1 - https://github.com/ttkb-oss/dedup/blob/6a906db5a940df71deb4f...

I wrote a small utility to find and dedup[0] using APFS features and had to poke around the user exposed metadata quite a bit. The short answer is probably. Metadata is extendable and there are things that work similarly already (encryption, and clone ids, for example). 0 - https://github.com/ttkb-oss/dedup

I just tested the C implementation on a utility I wrote[0] and at least on macOS where SHA256 is hardware accelerated beyond just NEON, BLAKE3 ends up being slower than SHA256 from CommonCrypto (the Apple provided crypto library). BLAKE3 ends up being 5-10% slower for the same input set. As far as I'm aware, Apple does not expose any of the hardware crypto functions, so unless what exists supports BLAKE3 and they add support in CommonCrypto, there's no advantage to using it from a performance perspective. The rust implementation is multithreaded and ends up beating SHA256 handily, but again, for my use case the C impl is only single threaded, and the utility assumes a single threaded hasher with one running on each core. Hashing is the bottleneck for `dedup`, so finding a faster hasher would have a lot of benefits. 0 - https://github.com/