open_clip

CLIP is a such a nice paradigm shift. Historically, CV things were quite limited: - You could predict a class (from a static list such as [dog,cat, ...]) or ... - You could use image embeddings disconnected from text (you could tell image look-alikes but not what they actually represent). By "embedding" text and images in the same latent space, you can now query your images with text query (such a "a large dog") and find the relevant photos. CLIP understands semantics but also is not limited to a set list of classes (thanks to the ability to use of web data in training). This is a list compiled by OpenCLIP of high performance models (some better than MetaCLIP) for those interested in using CLIP: https://github.com/mlfoundations/open_clip/blob/main/docs/op...

I welcome anyone who works at Adobe to simply answer this question and put it to rest. There is absolutely nothing sensitive about the issue, unless it exposes them in a lie. So no chance. I think it's a big fat lie. They'd have to have made some other scientific breakthrough, which they didn't. Using information from https://openai.com/research/clip and https://github.com/mlfoundations/open_clip, it's possible to answer this question. It's certainly not impossible, but it's impracticable. On 248m images (roughly the size of Adobe Stock), CLIP gets 37% on ImageNet, and on the 2000m from LAION, it performs 71-80%. And even with 2000m images, CLIP is substantially worse performing than the approach that Imagen uses for "text comprehension," which relies on essentially many billions more images and text tokens.

https://replicate.com/pharmapsychotic/clip-interrogator using: cfg.apply_low_vram_defaults() interrogate_fast() I tried lighter models like vit32/laion400 and others etc all are very very slow to load or use (model list: https://github.com/mlfoundations/open_clip) I'm desperately looking for something more modest and light.

Have you looked at LAION-400M? And the OpenCLIP [1] people have replicated CLIP performance using LAION-400M. [1] https://github.com/mlfoundations/open_clip