h2o-2

you should check out http://0xdata.com/ ; it's built from the ground up on a custom dkv to do in-memory ML. Reasons to check it out: 1 - it's open source https://github.com/0xdata/h2o 2 - ingest data from hdfs, s3, csv 3 - I've built systems like what you're discussing twice; the ML algorithms are often easier to write than expected while data management (moving data, sending updates, etc) which initially seems easier is much harder. 0xdata handles this for you. 4 - under active development 5 - it cleanly runs on your dev box with 1 or many nodes for development; deploying is a simple as uploading a jar to a cluster and putting a single file on each naming peers in the cluster 5a - see scripts to walk you through doing this disclosure: I work on it as of very recently =P

As far as I can tell, this is a different vision than that of Wakari. This effort seems to be about building a souped up parellel ML-oriented calculator around Hadoop. There is an R binding, there is a nascent Python binding (that "doesn't include the Python scientific frameworks because [they] are not that familiar with them"[1]. Wakari is an infrastructure for doing repeatable, collaborative analytics with Python. Our existing tools include a simple browser-based environment for code editing, IPython notebook, and sharing of plots and workflows. Our goal is to support multiple applications on that infrastructure, and not necessarily all written by Continuum. [1] https://github.com/0xdata/h2o/blob/master/py/h2o/h2o/cloud.p...

Innaresting. There was some blogging last year about consistency vs. availability issues, similar to what akka, riak, cloud Haskell etc are confronting. It occurs to me you need to do Bayesian inference to order packets on partition http://www.cliffc.org/blog/2012/05/ http://www.cliffc.org/blog/2012/06/21/progress-vacation/ ___________ very readable code, also, no generics in the few files i looke at. And, like twitter, scalaz, akka etc they write their own Futures https://github.com/0xdata/h2o/blob/8f6d81abeea8ef3221969d8aa...