MHmhx77·Nov 30, 2020
Yes, and I've documented this now:
https://github.com/mhx/dwarfs/blob/main/doc/dwarfs.md#settin...
In DwarFS: A fast high compression read-only file systemmhx
dwarfs
C++2020–2025
mhx / dwarfs
dwarfs
A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows
12Distinct people
19Mentions
0Explicit recommenders
2.6KGitHub stars
What to know before exploring
What it is
A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 2:59:58 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year19 dated mentions
Observed mentions: 2020-11-29 16:20:30 UTC to 2025-07-09 03:36:39 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 5 | 1 | 4 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 4 | 1 | 3 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 8 | 1 | 7 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
5 sourcesThis is really cool, I'll give squashfs-tools-ng a try!
> Does the Raspbian image comparison compare XZ compression against SquashFS with Zstd?
That's correct. It's not an exhaustive matrix of comparisons.
> Also, is there any documentation on how the on-disk format for DwarFS and it's packing works which might explain the incredible size difference?
The format as of 0.2.0 is actually quite simple. It's a list of compressed data blocks, followed by a metadata block (and a schema describing the metadata block). The metadata format is implemented by and documented in in [1].
There are probably 3 things that contribute to compression level:
1) Block size. DwarFS can use arbitrary block sizes (artificially limited to powers of two), and uses a much larger block size (16M) by default. SquasFS doesn't seem to be able to go higher than 1M.
2) Or
Read the rest
dering files by similarity.
3) Segment deduplication. If segments of files overlap with previously seen data, these segments are referenced instead of written again. The minimum size of these segments can be configured and defaults to 2k. For my primary use case, of the 47.6 GB of input data, 28.2 GB are saved by file-level deduplication, and another 12.4 GB by this segment-level deduplication. So before the "real" compression algorithms actually kick in, there are only 7 GB of data left. As these are ordered by similarity, and stored in rather big blocks, some of the 16M blocks can actually be compressed down to less then 100k.
[1] https://github.com/mhx/dwarfs/blob/main/thrift/metadata.thri...
Read-only media, for example. Or in general, stuff that doesn't really change. In my case: https://github.com/mhx/dwarfs#history
In DwarFS: A fast high compression read-only file systemWhew! It was easy to find out how you actually initialize this thing, if it's read-only:
https://github.com/mhx/dwarfs/blob/main/man/mkdwarfs.md
In DwarFS: A fast high compression read-only file systemDwarFS: A fast high compression read-only file system
280 points · Open discussion