GEgeekymartian·Jul 23, 2013
Nice lib, you may take some ideas from pismo: https://github.com/peterc/pismo , more metadata oriented but returns a nokogiri doc as well.
In Upton: A Web Scraping Frameworkpeterc
pismo
Ruby2010–2013
peterc / pismo
pismo
Extracts machine-readable metadata and content from Web pages
4Distinct people
7Mentions
0Explicit recommenders
747GitHub stars
What to know before exploring
What it is
Extracts machine-readable metadata and content from Web pages
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year7 dated mentions
Observed mentions: 2010-03-26 18:06:20 UTC to 2013-07-23 13:16:36 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2010 | 2010-01-01 00:00:00 UTC | 2011-01-01 00:00:00 UTC | 2 | 1 | 1 |
| 2011 | 2011-01-01 00:00:00 UTC | 2012-01-01 00:00:00 UTC | 4 | 0 | 4 |
| 2012 | 2012-01-01 00:00:00 UTC | 2013-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2013 | 2013-01-01 00:00:00 UTC | 2014-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
7 sourcesNice! I worked on something similar a year ago but for the Ruby world. If you're on a Unix with Ruby installed (e.g. OS X), you can mostly repeat the linked demo like so:
gem install pismo
pismo http://techcrunch.com/2011/06/09/twitter-ios/ title lede author datetime body
And then enjoy the output. No image pick out, but it's the first IMG tag in the 'html_body'.. just never got around to implementing it as I didn't need that feature.
The downside is I haven't worked on it for months and it's in sore need of improvements. For its current in-production use though, it's proving sufficient and a reasonable option for Rubyists. More info at https://github.com/peterc/pismo
Not knocking Jim's work on Plush, btw, he's actively working on it so if Java works out for you, stick to him! :)
In Instapaper like article extractor demo now onlineI created something similar in Ruby: https://github.com/peterc/pismo
In Instapaper like Article Extractor open sourcedCurrently in the middle of a re-architecture/re-write due to its flaws but something similar to this in Ruby I worked on last year: http://github.com/peterc/pismo
In Overview of Text Extraction AlgorithmsDiffbot's stuff is in a different league (but it's a hosted service with a large dataset) but if anyone's vaguely interested in this area, I've been working on a Ruby library that performs some similar features: https://github.com/peterc/pismo
It is currently undergoing the "big rewrite" (which includes some proper classification work rather than shooting in the dark), however, but it's still in daily use on several sites. Hopefully I can learn a few lessons from Diffbot!
I should also point out BoilerPlate - http://code.google.com/p/boilerpipe/ - an interesting Java based content extraction project that's being worked on an by an actual PhD student rather than a dilettante like me ;-) Again, Diffbot's stuff goes a lot further than this but there are lessons to be learned nonetheless.
Last but not least, a paper by the aforementioned PhD
Read the rest
student called Boilerplate Detection using Shallow Text Features is available at http://www.l3s.de/~kohlschuetter/boilerplate/
I suspect that there's going to be a lot more work in these areas in the medium term because of the growth of the "e-discovery" market and because the dreams of a consistently marked up "semantic Web" have been washing down the pan for a while now.
Great, i searched for a readability's ruby port on github but the closest i can find is pismo: https://github.com/peterc/pismo
In Ever wanted arc90′s Readability as an API?Pismo: extracts machine-readable metadata and content from Web pages
8 points · Open discussion