Qwen3-VL

Yes, it is possible to do those things and there are benchmarks for testing multimodal models on their ability to do so. Context length is the major limitation but longer videos can be processed in small chunks whose descriptions can be composed into larger scenes. https://github.com/JUNJIE99/MLVU https://huggingface.co/datasets/OpenGVLab/MVBench Ovis and Qwen3-VL are examples of models that can work with multiple frames from a video at once to produce both visual and temporal understanding https://huggingface.co/AIDC-AI/Ovis2.5-9B https://github.com/QwenLM/Qwen3-VL

I didn't understand what is meant by "pinpoint nearly every detail". The article is titled with that but then firehoses a bunch of technical details. The github spells it out much better: https://github.com/QwenLM/Qwen3-VL?tab=readme-ov-file#cookbo...

As far as I am aware, the "hanging" issue remains unsolved to this day. The underlying problem is that LLMs sometimes get stuck in a loop where they repeat the same text again and again until they reach the token limit. You can break the loop by setting a repeat penalty, but when your image contains repeated text, such as in tables, the LLM will output incorrect results to prevent repetition. Here is the corresponding GitHub issue for your default model (Qwen2.5-VL): https://github.com/QwenLM/Qwen2.5-VL/issues/241 You can mitigate the fallout of this repetition issue to some degree by chopping up each page into smaller pieces (paragraphs, tables, images, etc.) with a page layout model. Then at least only part of the text is broken instead of the entire page. A better solution might be to train a model to estimate a heat map of character d

Yep, this is true. I was poking around on their github and they have examples in their “cookbooks” section. Eg: https://github.com/QwenLM/Qwen2.5-VL/blob/main/cookbooks/ocr...

Thank you for sharing this. Some of the other public models that we can host ourselves may perform in practice better than the models listed - e.g. Qwen 2.5 VL https://github.com/QwenLM/Qwen2.5-VL?tab=readme-ov-file

Visual language models are quite good at converting images to LaTeX or any other kind of representation these days. Here is a demo where you can upload a screenshot of a page with an equation and ask the neural network to e.g. "Transcribe this equation as LaTeX.": https://huggingface.co/spaces/Qwen/Qwen2-VL You can also run a smaller model locally if you have enough VRAM, for example Qwen2.5-VL-7B-Instruct: https://github.com/QwenLM/Qwen2.5-VL?tab=readme-ov-file#usin... Also works reasonably well with hand-written equations. For searching through similar equations, you can probably embed each as a high-dimensional vector and then search for the closed vector. Here is a ranking of text embedding networks: https://huggingface.co/spaces/mteb/leaderboard Or if you want something more deterministic, parse the LaTeX equation to create an abstra