unblob

When these tools come up, I like to post my little list of similar tools, for those interested. Apparently it's a hobby of mine: * fq - like jq for binary data: https://github.com/wader/fq * Kaitai Struct - https://kaitai.io/ * visualizer, for the above: https://github.com/kaitai-io/kaitai_struct_visualizer/ * HexFiend - a hex editor, but with "binary templates" feature : https://github.com/HexFiend/HexFiend * binary templates, for the above: https://github.com/HexFiend/HexFiend/blob/master/templates/T... * binspector - https://github.com/binspector/binspector * binary-parsing - a collection of links to similar such tools : https://github.com/dloss/binary-parsing * unblob - https://github.com/onekey-sec/unblob * ImHex - https://github.com/WerWolv/ImHex * GNU Poke - https://www.gnu.org/software/poke/ * dtrx - https://github.com/dtrx-py/dtr

I think it's more complicated. If you are interested in a real-world implementation, check out our ELF parser: https://github.com/onekey-sec/unblob/blob/main/unblob/handle...

Thanks! I haven't really looked at alternative carvers (but I am aware that many great options exist). I didn't know unblob yet. Thanks for the suggestion! Looking at the code, they use the same methodology as I describe in the post [0]. They do, however, also check for the end of the last segment, which is interesting. [0] https://github.com/onekey-sec/unblob/blob/112da43587c80cc3a7...

Just piggybacking here to mention a variety of other "interpret structured binary data" tools. Apparently I collect links to these (: * fq - like jq for binary data: https://github.com/wader/fq * Kaitai Struct - https://kaitai.io/ ** visualizer, for the above: https://github.com/kaitai-io/kaitai_struct_visualizer/ * HexFiend - a hex editor, but with "binary templates" feature : https://github.com/HexFiend/HexFiend ** binary templates, for the above: https://github.com/HexFiend/HexFiend/blob/master/templates/T... * binspector - https://github.com/binspector/binspector * binary-parsing - a collection of links to similar such tools : https://github.com/dloss/binary-parsing * unblob - https://github.com/onekey-sec/unblob * ImHex, which you mention

If you are interested in implementation details, how to unpack/decompress them, check out these Python implementations: - https://github.com/onekey-sec/unblob/blob/main/unblob/handle... - https://github.com/onekey-sec/unblob/blob/main/unblob/handle... - https://github.com/onekey-sec/unblob/blob/main/unblob/handle...

Oh, this is one I hadn't heard of. There are several different takes on this idea, and I've had the notion to make my own for some years, too (not enough free time...) I feel like a good "how powerful is this tool, really?" test would be to model the Adobe PSD format with it. That format is bananas, and is what got me into this topic in the first place. But for those interested, here are some other tools in this vein: * fq - like jq for binary data: https://github.com/wader/fq * Kaitai Struct - https://kaitai.io/ ** visualizer, for the above: https://github.com/kaitai-io/kaitai_struct_visualizer/ * HexFiend - a hex editor, but with "binary templates" feature : https://github.com/HexFiend/HexFiend ** binary templates, for the above: https://github.com/HexFiend/HexFiend/blob/master/templates/T... * binspector - https://github.com/binspector

hyperscan is supported on Intel 64 bit only, but there is another project wchich supports ARM called vectorscan. My colleague wrote a Python wrapper for vectorscan: https://github.com/vlaci/pyperscan and the initial work they already merged: https://github.com/onekey-sec/unblob/pull/475

The main difference is that binwalk goes through a file linearly, searching for patterns like magic bytes, and tries to extract everything it finds. The problems with this: - very noisy, finding a lot of false positives (license code, format inside another format, etc). - very slow, trying to extract irrelevant things - imprecise, because it finds patterns in the middle of a file, where it's actually not relevant on the first level of extraction unblob solves these problems by being smarter about the file formats, recognizing them by their specification, for example unpacks format header structs and carves out files based the information in the header (size, offset). See a simple example for NTFS [1]. We also went to great lengths preventing unnecessary work by skipping formats inside another [2]. We are using hyperscan [3] instead of gr

In our firmware extraction suite, we started searching patterns with Yara, but it was so slow we switched to Hyperscan: https://github.com/onekey-sec/unblob/blob/main/unblob/finder... We are quite happy with the huge performance boost, never looking back :) Yara was even segfaulted when we tried to extract an Ubuntu ISO image with it, never had problems like that with Hyperscan. The only problem with Hyperscan is that it only supports Intel CPUs (it has some hand-crafted assembly), so it doesn't work on Apple M1 Macs, but there is a fork called VectorScan, which is working on ARM: https://github.com/VectorCamp/vectorscan We implemented a couple of small DSL classes in Python to be able to define YARA-like rules, it might be useful for you too, check it out here: https://github.com/onekey-sec/unblob/blob/cdd7a46667ffdfdfae...