riko

I loved pipes. I had rss feeds from all the sites where I was sharing content collected up and formatted via pipes into a single rss feed that was pulled into a php blog. Then all those sites I used to post on stopped supporting rss one by one and finally pipes was killed off. For a while I used a python library called riko that did the same thing as pipes without the visual editor. I have to thank it for getting me off php and into python. https://github.com/nerevu/riko

Check out my project, riko. https://github.com/nerevu/riko Open source, but no hosted plan.

I had one of the most complex Pipes rigged up to automate the collection of all the content I created or favorited from all over the web. It merged them all into a single rss feed which my very simple bit of python code scraped and output as static html on my "blog". When yahoo killed off pipes I found https://github.com/nerevu/riko and realised that the visual aspect of pipes was holding me back. Having said that, the visual aspect of pipes was what made it easy for me to get into the whole idea of "stream processing". So I have a lot to thank the pipes team for.

This python project is modelled on yahoo pipes is worth taking a look at: https://github.com/nerevu/riko

Sure, I'd be happy to help. On viewing the html, I'm not sure what software stack you're using, but maybe check out the riko[1] library, which was recommended in a parent comment. [1] https://github.com/nerevu/riko

If I understand you correctly, you want to be able to join the results of multiple sources? You can do something like that with a library I wrote, riko [1]. A simple example would look something like this: from riko.modules import join, fetch, fetchdata rss_url = 'http://site.com/rss' json_url = 'http://site.com/json' json_path = 'path.to.data' fetch_conf = {'url': json_url, 'path': json_path} rss = fetch.pipe(conf={'url': rss_url}) json = fetchdata.pipe(conf=fetch_conf) joined = join.pipe(rss, other=json) next(joined) You can see the docs for the `join` pipe here [2]. [1] https://github.com/nerevu/riko [2] https://github.com/nerevu/riko/blob/master/riko/modules/join...

You can check out my library riko [1, 2]. While it doesn't have a slick GUI, it does support most of the original yahoo pipes (including xpath [3] and regex [4]). [1] https://github.com/nerevu/riko [2] https://www.youtube.com/watch?v=bpn2G3TAAYY [3] https://github.com/nerevu/riko/blob/master/riko/modules/xpat... [4] https://github.com/nerevu/riko/blob/master/riko/modules/rege...

I wrote riko [1] with exactly this use case in mind. Its a simple python stream processing engine. [1] https://github.com/nerevu/riko Edit: HN submission https://news.ycombinator.com/item?id=12136011

> We use somewhat different concepts. I tend to think of streams as infinite, so it didn't occur to me to include something like a reverse pipe operator. We are in agreement. reverse has a notice that it isn't lazy [1]. I prefer to include pipes that aren't lazy since it can be helpful in some cases (plus the goal is to include all pipes originally in Yahoo! Pipes). The vast majority of pipes work just fine on infinite streams [2]. > I'm a bit surprised, why is filter an operator rather than a processor? I would think filters usually apply per-item, not to a whole stream? Just an implementation detail [3]. I agree it would be better if it were a processor since it could be parallelized. PRs welcome :). > I haven't worked on it very much but I'm heading towards push-based, using 0MQ for distribution/parallel processing, and using asyncio,

Cool! I actually starred your project last year. Never really got around to looking under the hood though. How do you handle the "multiple destinations" part? In python you can do it with a coroutines [1, 2] implementation (push based). I avoided that, since it coroutines have their own form of callback hell, and decided to implement a generator api (pull based) [3, 4]. But since generators can only be "pulled" into one destination, you have to copy a stream (subsequently converting it into a list) if you want more than one destination [5]. This works fine if the data can fit in memory, but if it can't then you're out of luck! [1] http://www.dabeaz.com/coroutines/copipe.py [2] http://www.dabeaz.com/coroutines/ [3] http://www.dabeaz.com/generators/retuple.py [4] http://www.dabeaz.com/generators [5] https://github.com/nerevu/riko/blob/maste

I just read about dask earlier today, very neat project! riko already handles parallel processing [1] but adding distributed processing sounds tempting. TBH though, distribution isn't high on the priority list. But I'll be happy to accept a PR if you are so inclined :) [1] https://github.com/nerevu/riko#parallel-processing

Interesting project. I hadn't come across this one yet. One difference that riko has is it's based around functions whereas this library (and practically every stream processing lib I've come across) is based around classes. I personally prefer the functional approach much better. And if you compare the word count examples on the respective readmes [1, 2], you will see riko is much more succinct. But I suppose the verbosity of the other libraries come with benefits like scaling across a cluster of servers. [1] https://github.com/plecto/motorway#word-count-example [2] https://github.com/nerevu/riko#word-count

`riko` is pure python stream processing library for analyzing and processing streams of structured data. It's modeled after Yahoo! Pipes [1] and was originally a fork of pipe2py [2]. It has both synchronous and asynchronous (via twisted) APIs, and supports parallel execution (via multiprocessing). Out of the box, `riko` can read csv/xml/json/html files; create text and data based flows via modular pipes; parse and extract RSS/ATOM feeds; and bunch of other neat things. You can think of `riko` as a poor man's Spark/Storm... stream processing made easy! Feedback welcome so let me know what you think! Resources: FAQ [3], cookbook [4], and ipython notebook [5] Quickie Demo: >>> from riko.modules import fetch >>> >>> stream = fetch.pipe(conf={'url': 'https://news.ycombinator.com/rss'}) >>> item = next(stream) >>> item['tit

Check out riko [1]. It was developed to programmatically replicate Yahoo Pipes (without the UI). But as it stands, I'm pretty sure it can do most of what google feeds does. [1] https://github.com/nerevu/riko (full disclosure: I'm the lead dev.)