clip-as-service

There are a few things in this space that might be interesting: - https://github.com/Hironsan/bertsearch - https://github.com/hanxiao/bert-as-service There was another I can't find right now that looked more polished/professional. But, in short, no it's pretty easy to setup. Just need a machine with a pretty big disk and be ok with an index latency. If you're using elastic search then you're already there on both accounts!

When using BERT (and all the many things like it, such as earlier ELMO, ULMfit and later ROBERTA/ERNIE/ALBERTa/etc) as the 'embeddings' you provide as input all the tokens in a sequence. You don't get an "embedding for word foobar in position 123", you get an embedding for all the sequence at once, so whatever corresponds to that token is a 728-dimensional "embedding for word foobar in position 123 conditional on all the particular other words that were before and after it'. Including very long-distance relations. One of the simpler ways to try that out in your code seems to be running BERT-as-a-service https://github.com/hanxiao/bert-as-service , or alternatively the huggingface libraries that are discussed in the original article. It's kind of the other way around compared to word2vec-style systems; before that you used to have a 'thin'

Hey, you might want to consider bert-as-service[0] for deep feature extraction from a BERT model. It will give you a 768 dimensional representation of the description, then you can embed that in the 512-dim latent space? I've been thinking of something similar. It's not that hard to do it yourself, but it's a really clean package, and it gives you nice CLI flags for most things like pooling strategy, and what layer you want to get the activations from. [0] https://github.com/hanxiao/bert-as-service

This is true, but rapidly changing. In addition to fine tuneable language models, you can do deep feature extraction with something like bert-as-service [0] ... You can even fine tune Bert on your days, then use the fine tuned model as a feature extractor. [0] https://github.com/hanxiao/bert-as-service