MiniGPT-4

The Fuyu pre-trained model is not open source. At best, it is source-available. It's also not the only multimodal model you can run locally. A few other examples include LLaVA[0], IDEFICS[1][2], and CogVLM[3]. Mini-GPT[4] might be another one to look at. I'm pretty sure all of these have better licenses than Fuyu. Fuyu's architecture does sound really interesting, but the license on the pre-trained model is a complete non-starter for almost anything. [0]: https://github.com/haotian-liu/LLaVA [1]: https://huggingface.co/blog/idefics [2]: https://huggingface.co/HuggingFaceM4/idefics-80b-instruct [3]: https://github.com/THUDM/CogVLM [4]: https://github.com/Vision-CAIR/MiniGPT-4

I just think the tech has been out for so long it's not as big of a deal. Mini-Gpt4 has been out for 6 months! Of course the descriptions aren't exactly gpt-4 grade, but with mistral 7b being used as the language model instead of llama 7b, the reasoning ability will improve noticeably. [1] https://github.com/Vision-CAIR/MiniGPT-4

There's more info here: https://github.com/Vision-CAIR/MiniGPT-4 (Linked in the Readme of the repo.)

If I have to learn how to be able to read this code and understand what it and its dependencies are doing, where do I start? Is reading their paper an effective strategy? https://github.com/Vision-CAIR/MiniGPT-4/blob/main/MiniGPT_4...

Filed an issue: Up or down vote there as you see fit. https://github.com/Vision-CAIR/MiniGPT-4/issues/9