CLIP

>which describes images in natural language, See CLIP https://github.com/openai/CLIP

Do you think https://github.com/openai/CLIP can be ran on it? LLM makes me think of chatbots but I suppose because it's inference-based it would work. Somewhat unclear on what's the difference between LLMs and inference, I think inference is the type of compute LLMs use. I wonder if inference-d would be a fitting name.

Yes, you could implement image similarity search using embeddings: Create embeddings for the entire image set, save the embeddings in a database, and add embeddings incrementally as new images come in. To search for a similar image, create the embedding for the image that you are looking for and compute the cosine similarity between that embedding and the embeddings in your database. The closer the cosine similarity is to 1.0 the more similar the images. For choosing a model, the article mentions the AWS Titan multimodal model, but you’d have to pay for API access to create the embeddings. Alternatively, self-hosting the CLIP model [0] to create embeddings would avoid API costs. Follow-up question: Would the embeddings from the llama3.2-vision models be of higher quality (contain more information) than the original CLIP model? The llama v

Biggest missing feature for all these self hosted photo hosting is the lack of a real search. Being able to search for things like "beach at night" is a time saver instead of browsing through hundreds or thousands of photos. There are trained neural networks out there like https://github.com/openai/CLIP which are quite good.

Does it have search by keywords/semantics? That would be my main need. For example if I need to find photos of red algae I could just search for that. OpenAI open sourced CLIP a couple of years ago: https://github.com/openai/CLIP and I was planning to write something myself to index my vast photo library but got too lazy and gave up.

the first thing that comes to mind is CLIP: https://github.com/openai/CLIP maybe combining it with some sort of knowledge graph is what you're looking for

>I guess the image input is just too expensive to run or it's actually not as great as they hyped it. We already know they have a SOTA model that can turn images into latent space vectors without being some insane resource hog - in fact, they give it away to competitors like Stability. [0] My guess is a limited set of people are using the GPT-4 with CLIP hybrid, but those use-cases are mostly trying to decipher pictures of text (which it would be very bad at), so they're working on that (or other use-case problems). [0]https://github.com/openai/CLIP

Accidentally quadratic ! Byte pair encoding by construction is quadratic on the length of the words. And usually the input is pre-split into words before being given to the byte pair encoder. Hopefully they use something different implementation in prod. It needs to be sanitized against very long words (like 10k character long words :) ). In previous tokenizer like CLIP (https://github.com/openai/CLIP/blob/main/clip/simple_tokeniz... ) , they used additional preprocessing steps like html escaping and various cleanup preprocessing using some python library (ftfy, html and regex), which made porting the code exactly to other languages a real pain. Sadly this library doesn't solve that :'-(

> What's missing from these models is everything related to visual or spatial information (that is not encoded in text). I assume that there will be eventually be something like ChatGPT/InstructGPT where part of the input data is images and or videos, with and without captions. So it would have a way of connecting the language to the spatial (and temporal). Is that missing though? Dall-E is the next tab over. You can image to CLIP, and image to image is not just visual, it involves that language to spatial and visual step: "CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3." https:/

They released CLIP (both model and code[1]), which is very broadly used in Dall-E alternatives. For example Stable Diffusion uses it. They also release Whisper model and code[2] [1] https://github.com/openai/CLIP [2] https://github.com/openai/whisper

OpenAI announced and released CLIP on GitHub on January 5, 2021: https://github.com/openai/CLIP You need CLIP to have CLIP guided diffusion. So the current situation seems to trace back to OpenAI and the MIT-licensed code they released the day DALL-E was announced. I would love to be corrected if I've misunderstood the situation.

Thanks. The next step would be combining it with text-image foundation models such as clip https://github.com/openai/CLIP so that the model no longer depends on a limited set of predefined labels (coco…), right? Also occlusion inference would be fantastic, so that we can select between the visible parts of the object and the whole shape (behind trees etc). Exciting decade.

This uses OpenAI’s CLIP model which is open source: https://github.com/openai/clip

I agree having a good vector is important to start with. However this is not very hard to make it work, you only need to finetune some of the clip models[1] to run it well. Disclose: I have built a vector search engine to proof this idea[2] [1]: https://github.com/openai/CLIP [2]: https://unisearch.cc/search?q=how+to+reset+mac+ssd

Not sure if this is the same thing? https://github.com/openai/CLIP