Pack

on OS metadata. There will always be something missing, and Pack tends to get everything perfect or nothing. Storing metadata adds extra weight that most users don’t care about and complicates the ability to store other types of data alongside files. It may get added as an option in the future if many people need it. - There is a version field. It is currently in Draft 0, and it is written using a custom VFS. Look here for more information: https://github.com/PackOrganization/Pack/blob/main/Source/Dr... - All future versions of Pack must handle previous versions and must only write the latest version. So any files created right now (Draft 0) will be read correctly for ever to come. - Each Draft proposal will get its own version, and if it gets final, it will be set to final. - Two byte after 'Pack' header in little endian as (1 (Draft) shl 13 + 0 (version 0) = 8192). Final would be 0, so the first Final version will be 0 shl 13 + 1 = 1. and the second will be 2. It is by design, so any Draft version gets a higher number, preventing future mixups. - 8 MB chunks are the default; Pack may choose smaller or bigger (16 MB for many small files or 32 MB for Hard Press). - Random access is proper as unpacking steps take into account what you want and decompress a Content just once for many neighbouring files. But even for reading just one file, here is an example: if I want to extract just a .c file from the whole codebase of Linux, it can be done (on my machine) in 30 ms, compared to near 500 ms for WinRAR or 2500 ms for tar.gz. And it will just worsen when you count encryption. For now, Pack encryption is not public, but when it is, you can access a file in a locked Pack file in a matter of milliseconds rather than seconds.

e POSIXy IO APIs that undoubtedly most of the selected implementations use. It's slightly odd that it appears to perform better when SQLite is being built with thread safety disabled (https://github.com/PackOrganization/Pack/blob/main/Libraries...) and yet the implementation is inserting in a thread group: https://github.com/PackOrganization/Pack/blob/main/Source/Dr.... I suspect the answer here is that because the implementation is using a thread group to read files and compress chunks, it's amortizing the slow cost of file opens in this benchmark using threading, but is heavily constrained by the sqlite locking - and the compression ratio will take a substantial hit in some cases as a result of the limited range of each compression operation. I suspect that zstd(1) with -T0 would outperform this for speed and compression ratio, and it's already installed on a lot of systems - even Windows 11 gained native support for .zst files recently. The premise that we could do with something more portable than TAR and with less baggage is somewhat reasonable - we probably could do with a simple, safe format. There are a lot more key considerations to making such a format good, such as many you outline, such as choices around seeking, syncing, incremental updates, compression efficiency, parallelism, etc. There is no single set of trade-offs to cover all cases but it would be possible to make a file format that can be shared among them, while constraining the design somewhat for safety and ease of portability.