tesser

There is a library called tesser [1] (by Jepsen/Riemann author) that behaves like parallel transducers with more Clojure "native" syntax. With transducers, you have to use "comp," and with tesser, you use "->>" as you'd use with lazy functions. Sadly, tesser is not advertised as it should; I find it much more flexible than transducers. E.g. you could parallelize tesser code over Spark/Hadoop cluster. [1] https://github.com/aphyr/tesser

Within the standard library, `clojure.core/pmap` is good for simple problems[0], whereas the various functions in the clojure.core.reducers[1] namespace are a bit more sophisticated and would probably solve the sort of problem you're describing pretty well. There are also a number of good clojure parallelism libraries floating around if you don't mind incurring dependencies: I've used claypoole[2] and tesser[3] in the past and been pretty happy with them. All of these options are, IME, relatively easy to drop in to some extant data processing pipeline to parallelize it, and probably require a similar level of finagling to what you're used to in Haskell. [0] (->> some-lazy-seq (pmap ...) (reduce ...)) goes pretty far, but nesting/composing pmaps or doing i/o doesn't always work particularly well since the JVM [currently] uses OS threads ra

Looks superficially simmilar to https://github.com/aphyr/tesser anyone know both and can give a comparison? From a brief examination tesser looks a lot simpler (probably because of encoding most of the folding using various monoids). Does onyx have a similar abstraction model that I missed?

Well, it depends. I think the primary improvement is that loop is no longer a language construct. It is just another function. As a prime example I would take Clojure. On first look: "No loops?" "This reduce functions everyhere look ugly." On second look: "Oh, I can import paralel reduce instead of the single-threaded one?" [1] "Somebody created a library to transparently switch between local reduce and one using hadoop?" [2] [1] http://clojure.org/reducers [2] https://github.com/aphyr/tesser