tarindexer

The bioinformatics community uses block based gzip compression (bgzip) [0]. The gzip standard allows for blocks so, using an additional index file, you can use it to seek to arbitrary locations and uncompress the block. gzip compression is maybe not optimal now and the block segmentation reduces the efficiency even further. Though not very standard, there is also a tar indexer program [1] that allows you to create an index on tar files to do the same. My information is at least a couple years old so things may have changed. [0] http://www.htslib.org/doc/bgzip.html [1] https://github.com/devsnd/tarindexer

There exists a utility called tarindexer [0] that can be used for random access to tar files. An index text file is created (one time) that is used to record the position of the files in the tar archive. Random reads are done by loading the index file and then seeking to the location of the file in question. For random access to gzip'd files, bgzip [1] can be used. bgzip also uses an index file (one time creation) that is used to record key points for random access. [0] https://github.com/devsnd/tarindexer [1] http://www.htslib.org/doc/bgzip.html

> Dumb question: Is it possible to create a utility or even a hack that performs "random access" into tar archives? Yes. If the tar file is on a seek-able medium, just read the headers only and build an index to the offsets of the file contents from the header data. Then use the offsets index to seek to just the item of interest and read out only it and nothing more. Now, does such a utility already exist? The answer seems to be yes: https://github.com/devsnd/tarindexer

After writing one myself along with a guy on IRC who needed a 'quick way to access huge .tar files', I found one written in python: https://github.com/devsnd/tarindexer He ended up doing the smart thing and changing his archiving program to not use .tar files, which solved all of his problems, but for the 2TB of data he already had this worked rather well.