CLIP

>I guess the image input is just too expensive to run or it's actually not as great as they hyped it. We already know they have a SOTA model that can turn images into latent space vectors without being some insane resource hog - in fact, they give it away to competitors like Stability. [0] My guess is a limited set of people are using the GPT-4 with CLIP hybrid, but those use-cases are mostly trying to decipher pictures of text (which it would be very bad at), so they're working on that (or other use-case problems). [0]https://github.com/openai/CLIP

Accidentally quadratic ! Byte pair encoding by construction is quadratic on the length of the words. And usually the input is pre-split into words before being given to the byte pair encoder. Hopefully they use something different implementation in prod. It needs to be sanitized against very long words (like 10k character long words :) ). In previous tokenizer like CLIP (https://github.com/openai/CLIP/blob/main/clip/simple_tokeniz... ) , they used additional preprocessing steps like html escaping and various cleanup preprocessing using some python library (ftfy, html and regex), which made porting the code exactly to other languages a real pain. Sadly this library doesn't solve that :'-(