kaitai_struct

There's a metric ton of them by now. Here's incomplete notes from a couple of years ago: ### kaitai - https://github.com/kaitai-io/kaitai_struct - https://github.com/kaitai-io/awesome-kaitai - http://formats.kaitai.io/dos_datetime/index.html ### Hexinator / Synalyze It! - Universal Parsing Engine - Hexinator is freemium version of Synalyze It! - https://github.com/synalysis/Grammars/blob/master/bitmap.gra... ### quickbms - http://aluigi.altervista.org/quickbms.htm ## multiex - http://multiex.xentax.com/ ### Game Extractor by WATTO - http://www.watto.org/game_extractor.html ### 010 editor templates - https://www.sweetscape.com/010editor/repository/templates/ ### hex fiend templates - https://github.com/HexFiend/HexFiend/tree/master/templates ### malcat - has some form of binary templates - https://malcat.fr/ ### Andys Binary Fo

I contributed a number of file formats a few years ago (and attempted numerous others) but ran into a number of problems with certain file formats: 1. It's not possible to read from the file until a multiple byte termination sequence is detected. [1] 2. You can't read sections of a file where the termination condition is the presence of a sequence of bytes denoting the next unrelated section of the file (and you don't want to consume/read these bytes) [2] 3. The WebIDE at the time couldn't handle very large file format specifications such as Photoshop (PSD) [3] 4. Files containing compressed or encrypted sections require a compression/encryption algorithm to be hardcoded into Kaitai struct libraries for each programming language it can output to. The WebIDE I particularly liked as it makes it easy to get started and share results. I also

i found the Kaitai toolset (IDE, compiler) to be useful in parsing or "deserialization" for proprietary financial protocols. if you are just reading data (especially in an environment with clients in different languages) it's a strong recommend, however lack of support for "serialization" means that you will still need to roll your own encoders. ultimately we created our own tooling/DSL for encoding as well as generating KSY files to generate clients as part of our builds edit: i forgot to add - there is an issue for serialization on kaitai github repo for some time [1], with some interesting discussion around the implementation challenges 1. https://github.com/kaitai-io/kaitai_struct/issues/27