gensim

Nice work! I wonder if you're going the same challenges that gensim had for being generic in summarization. For context: > Despite its general-sounding name, the module will not satisfy the majority of use cases in production and is likely to waste people's time. https://github.com/RaRe-Technologies/gensim/wiki/Migrating-f...

> Problem statement: the Levenshtein distance is a string metric for measuring the difference between two sequences Another variant is "I have a bunch of words (a dictionary) and one query word, and want to find all words from the dictionary that are close to the query word". This leads to another interesting class of problems, because you can do clever things where you precompute search structures (e.g. Levenshtein automata [0], [1]) from the dictionary. The similarity queries then run (much) faster. In production, performance matters. We recently merged a PR like that into Gensim [2]. This gave a ~1,500x speed-up compared to naively comparing all pairwise strings with Levenshtein distance. A difference between the training step running for months (=unusable) and minutes. [0] http://blog.notdot.net/2010/07/Damn-Cool-Algorithms-Levensht..

Apparently it did: https://github.com/RaRe-Technologies/gensim/issues/1873

Gensim is #119 in the list according to your link, far behind projects with many more active contributors, so hardly a resounding success of your scoring method. In terms of metrics, you could start by weighing projects with few contributors as more critical, not less. Specifically, gensim does appear to have had quite a few contributors, but the bulk of the code was written by the single maintainer https://github.com/RaRe-Technologies/gensim/graphs/contribut... So maybe you should add a metric "percentage of code in the past year authored by the top contributor". If you want to go about it in a more data-driven fashion, you could go through the top projects for each language, check whether they actually need your support (e.g. find out what the development goals are, ask whether the current resources are sufficient and what they'd do wit

Timely! We just added a super-efficient implementation of Online Non-negative Matrix Factorization to Gensim. Jupyter notebook tutorial: https://github.com/RaRe-Technologies/gensim/blob/nmf_speedup... What's unique about this implementation is that it's streamed, i.e. can process arbitrary amounts of data in constant RAM. It's also heavily optimized for sparse inputs (~text corpora, not just dense images), just like the optimized sparse SVD, LDA, FastText etc in Gensim.

Gensim : https://github.com/RaRe-Technologies/gensim [Can't speak for the 'highest' part of the qn, but Gensim upholds very high code quality standards]

Note that like the article says, memory-mapping (mmap) vectors from disk already works in Gensim: https://radimrehurek.com/gensim/models/keyedvectors.html The low memory footprint in LMDB is interesting. There were also some exciting papers recently that used compression (vector quantization) [0]. We're still evaluating whether to add those to Gensim [1]. [0] word2bits, https://twitter.com/RadimRehurek/status/976015607941517312 [1] word2bits PR, https://github.com/RaRe-Technologies/gensim/pull/2011#issuec...

Radim: please consider incorporating this into gensim. It really is superior to simpler classification models running on top of word/BPE/wordpiece embeddings and to classic machine learning algorithms used for text classification and topic modeling like HDP, LDA, LSI/LSA, etc. (You can see for yourself how well this works out-of-the-box with a simple exercise: grab a pretrained model from fast.ai, run a bunch of documents through it, grabbing and saving each time the last hidden-layer representation of each document, and then map these representations to a two-dimensional plot with, say, t-SNE.) I realize that outside of Silicon Valley and other technology centers, most established companies are far -- far -- from adopting deep learning for any application of importance, due partly to the current unavailability of developers with AI exper

FYI: you can now use fastText directly from gensim (Python) [1]. This allows you to easily test / compare fastText to other popular embeddings, such as word2vec, doc2vec or gloVe. [1] https://github.com/RaRe-Technologies/gensim/blob/develop/gen...

Since you can basically write C in Cython, it depends how far you take the cython optimization. You can just add a few types, or you can go nuts via scipy's BLAS interface https://github.com/RaRe-Technologies/gensim/blob/develop/gen... In general, you can get to less overhead with Cython right now because it's a much more mature project and because you can be more flexible with defining the point where you drop from python to a faster alternative. I wouldn't use that Nim/Python lib in production unless you have time to help with its development. It's good for personal projects though.