redshift

Well, here's a less trivial example: https://github.com/syllog1sm/redshift/blob/develop/redshift/... The task/algorithm is roughly (but not precisely) the one described here: http://honnibal.wordpress.com/2013/12/18/a-simple-fast-algor... If you just scroll through parser.pyx, you might get the flavour of why this is better than straight C. The "core", or "skeleton" of the code is all C-typed, but I "fall-back" to Python for work-a-day stuff like reading and writing config files, collecting the set of labels, etc. There are also places in the algorithmic code where I use Python data structures. In Parser._count_feats, I use a dictionary of dictionaries. This would be quite annoying in C, and efficiency is quite irrelevant here. I think using Cython in this way has given me a competitive edge in my research. This library has by far the bes

Let me know if you're interested in licensing better syntactic/semantic parsing technologies. I'm the developer of the Redshift NLP library ( http://github.com/syllog1sm/redshift ). Currently the software lacks documentation, but it offers a good speed/accuracy trade-off. Documentation and a good tokenizer are coming. You can read a tutorial for a simplified version of the algorithm on my blog: http://honnibal.wordpress.com

Thanks for the explanation of how your cache works. Will you be at ACL this year? The memoisation I refer to is called here: https://github.com/syllog1sm/redshift/blob/segmentation/reds... What happens is, I extract the set of token indices for S0, N0, S0h, S0h2, etc, into a struct SlotTokens. SlotTokens is sufficient to extract the features, so I can use its hash to memoise an array of class scores. Cache utilisation is about 30-40% even at k=8. While I'm here... https://github.com/syllog1sm/redshift/blob/segmentation/reds... The big enum names all of the atomic feature values that I extract, and places their values into an array, context. So context[S0w] contains the word of the token on top of the stack. I then list the actual features as tuples, referring to those values. So I can write a group of features with something like new_feat

Found this one: https://github.com/syllog1sm/redshift

Accuracy is state-of-the-art -- 92-93% depending on the beam width and the evaluation set (Stanford or MALT dependencies). I assume that this is for English? A former colleague of mine compared two statistical dependency parsers (Malt and MST) to a rule-based dependency parser with a maxent disambiguation model, for Dutch. The rule-based system outperforms the statistical dependency parsers by a wide margin, both in-domain and out-of-domain: http://dl.acm.org/citation.cfm?id=1870171 Nonetheless, I find work on statistical dependency parsing to be very exiting, since it is fast and requires far less human effort :). https://github.com/syllog1sm/redshift/ . You'll want the develop branch. It's GPL licensed. Very nice work!

The Stanford parser is particularly slow --- it's in java, and it's written for research more than anything. The C&C CCG parser runs at about 60-80 sentences a second, although it gives either CCG constituents or dependencies -- so the output may take some interpretation. Shift-reduce dependency parsers are linear time, and are giving state-of-the-art results. My parser's currently a pain in the ass to install, as it hasn't really been released yet, but it does hundreds of sentences a second. Accuracy is state-of-the-art -- 92-93% depending on the beam width and the evaluation set (Stanford or MALT dependencies). https://github.com/syllog1sm/redshift/ . You'll want the develop branch. It's GPL licensed. It's implemented in Cython (i.e., almost all the code is Cython --- I'm not using it just for the speed critical bits), which would make