magnitude

I agree, and not just for text classification. Its a good baseline for anything that looks like text (if you train a model from scratch), e.g., URLs with some basic preprocessing. This is because FastText uses character n-grams; the consequence of this is substring similarities are exploited. Some things to note: (1) FastText comes with a classifier but I have often seen good results with using a difference classifier like LightGBM or a SVM. (2) If you want other word embeddings to compare with as well, magnitude [1] is an easy to use library. (3) FastText now supports multiple languages [2]. [1] https://github.com/plasticityai/magnitude#pre-converted-magn... [2] https://fasttext.cc/docs/en/crawl-vectors.html#models

Check out Magnitude, we built it to solve that problem: https://github.com/plasticityai/magnitude It's still loaded from a file, but heavily uses memory-mapping and caching to be speedy and not overload your RAM immediately. And in production scenarios, multiple worker processes can share that memory due to the memory mapping. Granted it's read-only, so might not be exactly what you are looking for. Disclaimer: I'm the author.

Our startup made a package powered by SQLite for this very purpose: https://github.com/plasticityai/magnitude Might be worth checking out :)