Read the discussion on HN. A full-thread summary is not available yet.
Zygo
bees
C++2020–2026
Zygo / bees
bees
Best-Effort Extent-Same, a btrfs dedupe agent
7Distinct people
8Mentions
0Explicit recommenders
1.1KGitHub stars
What to know before exploring
What it is
Best-Effort Extent-Same, a btrfs dedupe agent
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:03:16 PM UTC.
Mentions over time
HN mentions by year8 dated mentions
Observed mentions: 2020-09-29 04:14:21 UTC to 2026-01-02 14:43:09 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 3 | 0 | 3 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 2 | 0 | 2 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
8 sourcesEMembedding-shape·Jan 2, 2026Submission
Bees – Best-Effort Extent-Same, a btrfs dedupe agent
2 points · Open discussionIncidentally, the application I'm reworking to use btrfs is cocalc.com. One of our main use cases is distributed assignments to students in classes, as part of the course management functionality. Imagine a class with 1500 students all getting an exact copy of a 50 MB folder, which they'll edit a little bit, and then it will be collected. The copy-on-write functionality of btrfs is fantastic for this use case (both in speed and disk usage).
Also, the out-of-band deduplication for btrfs using https://github.com/Zygo/bees is very impressive and flexible, in a way that ZFS just doesn't match.
In Bcachefs Goes to "Externally Maintained"I've used bees successfully. https://github.com/Zygo/bees
In HyperspaceInstead of dupremove, bees may be more appropriate for your goals.
https://github.com/Zygo/bees
Bees does things at a lower level than just reflinking full files like dupremove (it dedups at the level of extents). Be sure to read the links in the "Recommended Reading" section, of the above link, to hopefully avoid issues.
In Tell HN: Deduplicating a 10.4 TiB game preservation archive (WIP)Which tool did you end up using for btrfs? I tried out bees https://github.com/Zygo/bees but it is way too slow.
In OpenZFS deduplication is good now and you shouldn't use itThe main problem is that you don't have the time and energy to go through 23TB of files. You want a simple solution to pack them all in a safe place and sort them later (which will never happen, but it's ok, we all have this problem).
Here's what to do:
1) Get a few very large HDDs, the largest you deem afordable. In total, you should have about 4x - 8x the capacity of your current files.
2) Split them in two groups: main storage and backup storage.
3) On each group, create a btrfs RAID1 and mount it with compression. Start with a btrfs raid1c3 if you can afford 3 drives, and downgrade to raid1 if you run out of space.
4) Copy all your files there. You may want to make a directory for each media you copy data from. You should extract all archives that you may already have created and let btrfs deal with compression - this allows for dedup
Read the rest
lication.
5) Run a deduplication tool that supports btrfs (https://github.com/Zygo/bees). Creating btrfs with sha256 checksums is probably better for block deduplication while using crypto acceleration available on most systems.
Safety rules:
- Only connect the backup drives when you do the backup. Keep the backup in the closet or, preferably, in another building (in case of fire, etc.).
- Label the drives. You don't want to accidentally mix the two groups.
Later, when you run out of space:
- buy a pair of even bigger drives and add them to the btrfs volume
- remove the old drives from the volume (this step is very time consuming)
or
- have a huge case or hdd rack and keep adding drives without removing any
One last thing: you may want to use a NAS or DAS, at least for the backup group.
I use btrfs + bees (https://github.com/Zygo/bees) for such cases
In Should you remove duplicate files?Sure, although the proper way to test it would be to write a lot of data to the drive, issue an fsync, and cut power in the middle of the operation. Rinse and repeat a (few) hundred times for each drive.
There's a guy on btrfs' LKML (also the author of [0]) who is diligent enough to do these tests on much of the hardware he gets, and his experience does not sound good for consumer drives.
[0]: https://github.com/Zygo/bees/
In Durability: NVMe Disks