restruct

ion from a design PoV; many things that would be calculated for parsing structures in deserialization could just become checks/assertions in serialization. As an example, checking that an expression calculates out to the expected value would be a reasonable approach. Reversible expressions could be implemented for some cases, too, if you want it to do more of the heavy lifting. I think the biggest obstacle is actually implementing it, and frankly my Scala is too weak to help with such a relatively big undertaking. I’ve also played with the rust nom library, which implements functional programming style parser combinators. It is quite cool how it can express fairly complex grammars and binary formats pretty much equally well, albeit optimizing it effectively requires serious magic that I do not think nom has. (I assume in Haskell, the same thing can be done with mind-boggling optimization power.) It also doesn’t handle serialization, but it’s still worth noting for robust deserialization. [1]: https://github.com/go-restruct/restruct [2]: https://github.com/jchv/pngextract [3]: https://github.com/jchv/flsplit

so ended up contributing improvements to Kaitai’s Go compiler too, for good measure. The bottom line is, these tools make it extremely easy to work with complex binary files. I can incrementally sketch out an unknown file format, like FL Studio FLP, and quickly test my assumptions about structure shape. Kaitai also has their IDE tool, which lets you use Kaitai interactively: https://ide.kaitai.io/ I actually hope to do the same for Restruct, but want to re-engineer some aspects of it to make it work better in an interactive context.

nts something really cool. A simple reflection-based mechanism that can read and write binary data into a structure in a defined and simple way. lunixbochs extended this to struc[1], which adds additional tags for advanced reading and writing of binary structures, including variable length structures. I went further and maybe a bit off into the deep end with Restruct[2], a similar concept but with a lot more features, designed specifically so I could handle advanced structures quickly. The end result is that I can define some Go structs with integers, strings, byte arrays and corresponding tags, and be able to serialize and deserialize from those structures to their corresponding binary representation. For an overdone demo of what you could do with Restruct for example, see this (incomplete) PNG demo: https://github.com/go-restruct/restruct/blob/master/formats/... (It is mainly incomplete because I had moved focus to develop a codegen for restruct, to improve runtime performance, although such work has since stalled.) Most of my inspiration for newer advanced features like expressions comes from Kaitai Struct[3], an excellent project I have also contributed to a bit. I learned about Kaitai after having written Restruct thanks to a friend, though its lack of write capabilities has lead me to continue using Restruct for many hacks, such as some tools I wrote that modified FL Studio Project files. [1]: https://pkg.go.dev/github.com/lunixbochs/struc [2]: https://pkg.go.dev/github.com/go-restruct/restruct [3]: https://kaitai.io