ratarmount

/blob/master/zlib/examples/... All depends on the use case of course. Seems like the author here has a pretty specific one - though I still don't see what the advantage of this is vs extracting in JS and adding all files individually to memfs. "Without any copying" doesn't really make sense because the only difference is copying ONE 1MB tar blob into a Uint8Array vs 1000 1kB file blobs One very valid constraint the author makes is not being able to touch the source file. If you can do that, there's of course a thousand better solutions to all this - like using zip, which compresses each file individually and always has a central index at the end.

SquashFS, fsspec, and many more backends have been added, so that it now should be able to read every format that archivemount can and some more, and even read them remotely. However, performance for any use case besides bzip2/gzip-compressed TARs may vary even though I did my best. Personally, I am using it view to packed folders with many small files that do not change anymore. I pack these folders because else copying to other hard drives takes much longer. I'm also using it when I want to avoid the command line. I have added ratarmount as a Caja user script for mounting via right-click. This way, I can mount an archive and then copy the contents to another drive to effectively do the extraction and copying in one step. Initially, I have also used it to train on the ImageNet TAR archive directly. I probably should have released a 1.0.0 some years ago because I have kept the command line interface and even the index file format compatible as best as possible between the several 0.x versions already. Some larger future features on my wishlist are: - A new indexed_lz4 backend. This should be doable inside my indexed_bzip2 [1] / rapidgzip [2] backend library. - A custom ZIP and SquashFS reader accelerated by rapidgzip and indexed_bzip2 to enable faster seeking inside large files inside those archives. - I am eagerly awaiting the Linux Kernel FUSE BPF support [3], which might enable some further latency reductions for use cases with very small files / very small reads, at least in the case of working with uncompressed archives. I have done comparisons for such archives (100k images a 100 KiB) and noticed that direct access via the Python library ratarmountcore was roughly two times faster than access via ratarmount and FUSE. Maybe I'll even find the time to play around with the existing unmerged FUSE BPF patch set. [0] https://news.ycombinator.com/item?id=30631387 [1] https://news.ycombinator.com/item?id=31875318 [2] https://news.ycombinator.com/item?id=37378411 [3] https://lwn.net/Articles/937433/

- [5] https://github.com/mxmlnkn/ratarmount

website comparing this. It says "Such behavior is very dangerous, as application might see wrong files in the /proc filesystem if it will try to access one via its PID." Now I was wondering if we could have a general sort of Virtual Machine which could run a specific language (especially the erlang , especially erlang with its let it fail approach could be nice) / For decompression software as well. Is criu the best we can get within the linux world , has somebody thought of creating a very lightweight os meant to be as a virtual machine with checkpoint / restore. Or could we rather instead of Criu , actually try to do this without Criu , right in the decompression software tooling ? I am somewhat confused about this , and I would really like it if someone with experience could simplify where I should try to put my efforts into doing such project

slightly faster than cat when comparing the piped bandwidth! E.g., compare these additional benchmarks: time cat 4GiB-base64.gz | wc -c # real ~1.06 s -> ~3.1 GB/s time fcat 4GiB-base64.gz | wc -c # real ~0.41 s -> ~8.0 GB/s time rapidgzip -o /dev/null -d 4GiB-base64.gz # real ~0.68 s -> ~6.5 GB/s fcat is an alternative cat implementation that uses vmsplice to speed up piping. According to the ReadMe it currently is broken, but it works fine on my system and piping it to md5sum yields consistent results [2]. So, at this point, I/O and actually also allocations have become a limiting factor and if you want full speed, you would have to interface with the rapidgzip library interface directly (in C++ or via the Python bindings) and process the decompressed data in memory. The project ReadMe contains further benchmarks with Silesia and FASTQ data and scaling up to 128 cores, for which rapidgzip achieves 12 GB/s for Silesia and 24 GB/s when an index has been created with --export-index and is used with --import-index. It can also be tested with ratarmount 0.14.0, which now uses rapidgzip as a backend by default for .gz and .tar.gz files [3]. [0] https://news.ycombinator.com/item?id=32366959 [1] https://dl.acm.org/doi/10.1145/3588195.3592992 [2] https://github.com/mre/fcat [3] https://github.com/mxmlnkn/ratarmount

offer random access to TAR files that is magnitudes faster than archivemount: https://github.com/mxmlnkn/ratarmount