magika

Gmail uses Magika https://github.com/google/magika to deduce file types to determine whether and how to scan email attachments.

https://github.com/google/magika Magika is a novel AI powered file type detection tool that relies on the recent advance of deep learning to provide accurate detection. It's not a sliver bullet, but I use it recently for inspecting file type instead of magic file. One advantage is that detecting composed files. Take pdf+exe file for example, the library will report something like 70% pdf and 30% pebin file.

Nice aka perfect timing.I just restored "some" files (40GB) with [1] But the filetype detection of photorec set some wrong file types. Edit: It would be super helpful if the "suffix" could be added as output so i can move the files to the right directory [2] ;) [1] https://www.cgsecurity.org/wiki/PhotoRec [2] https://github.com/google/magika/issues/63

I wonder how it performs with detecting C vs C++ vs ObjC vs ObjC++ and for bonus points: the common C/C++ subset (which is an incompatible C fork), also extra bonus points for detecting language version compatibility (e.g. C89 vs C99 vs C11...). Separating C from C++ and ObjC is where the file type detection on Github traditionally had problems with (but has been getting dramatically better over time), from an "AI-powered" solution which has been trained on the entire internet I would expect to do better right from the start. The list here doesn't even mention any of those languages except C though: https://github.com/google/magika/blob/main/docs/supported-co...

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/google/magika/blob/120205323e260dad4e5877... It loads the model an runtime: https://github.com/google/magika/blob/120205323e260dad4e5877... They mark it as Experimental in the documentation, but it seems like it was just made for the web demo. Also as others have mentioned. The model appears to only detect 116 file types: https://github.com/google/magika/blob/120205323e260dad4e5877... Where libmagic detects... a lot. Over 1600 l

Supported file types: https://github.com/google/magika/blob/main/docs/supported-co...

We are very excited to announce the release of Magika our AI powered fast and efficient file type identification lib and tool - https://opensource.googleblog.com/2024/02/magika-ai-powered-fast-and-efficient-fil… Thanks to its optimized Keras model, large scale training dataset, and Onnx Magika massively outperform other file identification tools while be very fast even on CPU. Magika python code and model is open sourced on Github: https://github.com/google/magika and we also provide an experimental TFJS based npm package https://www.npmjs.com/package/magika With the team we hope you will find Magika useful for your own projects. Let us know what you think or if you have any question!