magnitude

Interesting... looks like this uses pymagnitude https://github.com/plasticityai/magnitude

I agree, and not just for text classification. Its a good baseline for anything that looks like text (if you train a model from scratch), e.g., URLs with some basic preprocessing. This is because FastText uses character n-grams; the consequence of this is substring similarities are exploited. Some things to note: (1) FastText comes with a classifier but I have often seen good results with using a difference classifier like LightGBM or a SVM. (2) If you want other word embeddings to compare with as well, magnitude [1] is an easy to use library. (3) FastText now supports multiple languages [2]. [1] https://github.com/plasticityai/magnitude#pre-converted-magn... [2] https://fasttext.cc/docs/en/crawl-vectors.html#models

Check out Magnitude, we built it to solve that problem: https://github.com/plasticityai/magnitude It's still loaded from a file, but heavily uses memory-mapping and caching to be speedy and not overload your RAM immediately. And in production scenarios, multiple worker processes can share that memory due to the memory mapping. Granted it's read-only, so might not be exactly what you are looking for. Disclaimer: I'm the author.

Our startup made a package powered by SQLite for this very purpose: https://github.com/plasticityai/magnitude Might be worth checking out :)

See my top-level reply about a similar library called Magnitude that we have built and open-sourced: https://github.com/plasticityai/magnitude. It actually utilizes an LRU cache (configurable cache size argument on the constructor) so you can utilize an in-memory LRU cache as you query words off-disk using SQLite indexed disk-seeks. And you're right, due to Zipf-ian properties of most text, you can see gains even with a small in-memory cache size :).

This is interesting work. We at Plasticity (YC S17) have open sourced something similar called Magnitude (https://github.com/plasticityai/magnitude) a few months ago for querying vector embeddings quickly with low memory usage using SQLite and a standard universal file format (.magnitude) between word2vec, fastText, and GloVe. We also added features for out-of-vocabulary lookups, bulk queries, concatenating embeddings from multiple models, etc. It may also be of interest to folks looking at this. We also have a paper on Magnitude we will be presenting at EMNLP 2018 and ELMo support is coming soon!

We recently open sourced a library that makes it easy to quickly get started with pre-trained word vector models like word2vec from Google, Facebook, and Stanford. It adds a ton of extra functionality like fast similiarity indexing with Annoy and fairly robust out-of-vocabulary word lookups (handling misspellings) out of the box. If you want to quickly get started with pre-built models, it might be worth checking out along with Gensim: https://github.com/plasticityai/magnitude/

We recently open sourced a library that does exactly this: https://github.com/plasticityai/magnitude For word2vec, GloVE, and fastText. It is able to generate vectors for out-of-vocabulary words through "fragments" of words or subword character n-grams rather.

We just open sourced a easy to use library called Magnitude that handles out-of-vocabulary words and uses Annoy indexing for fast most_similar queries for word2vec, GloVE, and fastText: https://github.com/plasticityai/magnitude

We've recently created an open source library that can help you get started with pre-trained word vectors and Annoy quickly: https://github.com/plasticityai/magnitude