open_clip

Here’s some code: https://github.com/mlfoundations/open_clip?tab=readme-ov-fil...

The part of CLIP[1] that you need to know to understand this is that it embeds text and images into the same space. ie: the word "dog" is close to images of dogs. Normally this space is a high dimensional real space. Think 512-dimensional or 512 floating point numbers. When you want to measure "closeness" between vectors in this space cosine similarity[2] is a natural choice. Why would you want to quantize values? Well, instead of using a 32-bit float for each dimension, what if you could get away with 1-bit? You would save you 31x the space. Often you'll want to embed millions or billions of pieces of text or images, so the savings represent a huge speed & cost savings and if accuracy isn't impacted too much then it could be worth it. If you naively clip the floats of an existing model, it severely impacts accuracy. However, if you train

I have successfully used OpenCLIP models for embedding and similar-image search. The smallest model listed on that UForm page is 79 million parameters, so I presume that you can use other models of similar size. There are a few OpenCLIP models with 80 million or fewer parameters listed here: https://github.com/mlfoundations/open_clip/blob/main/docs/mo... When embeddings are quantized to int8 they still work very well for similarity (no differences in top 10 search on my test set). I haven't tried quantizing the models themselves.

It is a misconception that Adobe's models have not been trained on copyrighted work. Nobody should be repeating their marketing claims. Adobe has not shown how they train the text encoders in Firefly, or what images were used for the text-based conditioning (i.e. "text to image") part of their image generation model. They are almost certainly using CLIP or T5, which are trained on LAION2b, an image dataset with the very problems they are trying to address, C4 (a text dataset similarly encumbered) and similar. I welcome anyone who works at Adobe to simply answer this question of how they trained the text encoders for text conditioning and put it to rest. There is absolutely nothing sensitive about the issue, unless it exposes them in a lie. So no chance. I think it's a big fat lie. They'd have to have made some other scientific breakthroug

CLIP is a such a nice paradigm shift. Historically, CV things were quite limited: - You could predict a class (from a static list such as [dog,cat, ...]) or ... - You could use image embeddings disconnected from text (you could tell image look-alikes but not what they actually represent). By "embedding" text and images in the same latent space, you can now query your images with text query (such a "a large dog") and find the relevant photos. CLIP understands semantics but also is not limited to a set list of classes (thanks to the ability to use of web data in training). This is a list compiled by OpenCLIP of high performance models (some better than MetaCLIP) for those interested in using CLIP: https://github.com/mlfoundations/open_clip/blob/main/docs/op...

I welcome anyone who works at Adobe to simply answer this question and put it to rest. There is absolutely nothing sensitive about the issue, unless it exposes them in a lie. So no chance. I think it's a big fat lie. They'd have to have made some other scientific breakthrough, which they didn't. Using information from https://openai.com/research/clip and https://github.com/mlfoundations/open_clip, it's possible to answer this question. It's certainly not impossible, but it's impracticable. On 248m images (roughly the size of Adobe Stock), CLIP gets 37% on ImageNet, and on the 2000m from LAION, it performs 71-80%. And even with 2000m images, CLIP is substantially worse performing than the approach that Imagen uses for "text comprehension," which relies on essentially many billions more images and text tokens.

https://replicate.com/pharmapsychotic/clip-interrogator using: cfg.apply_low_vram_defaults() interrogate_fast() I tried lighter models like vit32/laion400 and others etc all are very very slow to load or use (model list: https://github.com/mlfoundations/open_clip) I'm desperately looking for something more modest and light.

Have you looked at LAION-400M? And the OpenCLIP [1] people have replicated CLIP performance using LAION-400M. [1] https://github.com/mlfoundations/open_clip

> Writing a training loop for CLIP manually wound up with me banging against all sorts of strange roadblocks and missing bits of documentation, and I still don't have it working. There is working training code for openCLIP https://github.com/mlfoundations/open_clip But training multi-modal text-to-image models is still a _very_ new thing, in terms of the software world. Given that, my experience has been that it's never been easier to get to work on this stuff from the software POV. The hardware is the tricky bit (and preventing bandwidth issues on distributed systems). That isn't to say that there isn't code out there for training. Just that you're going to run into issues and learning how to solve those issues as you encounter them is going to be a highly valuable skill soon. edit: I'm seeing in a sibling comment that you're hoping to

What sort of compute do you get access to? There's a lot of cool stuff you could do depending on whether or not you have decent GPU's and for how much time you're allowed to experiment on them. Experimentation is fairly fundamental in practice. There are a lot of pretraining tasks in vision/multimodal that are cool. Largely techniques introduced or refined by OpenAI re-implemented as pytorch open source codebases with varying degrees of success: - Finetune your own CLIP https://github.com/mlfoundations/open_clip - Train a (much smaller) DALLE https://github.com/lucidrains/DALLE-pytorch - Train your own guided diffusion https://colab.research.google.com/drive/1javQRTkALBWLFWnx1K4... (pretty tough, may only be feasible on domain-specific data) - Train a variational autoencoder (VAE) - "VQGAN" from Heidelberg https://github.com/CompVis/tamin