HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
node-js-libs
node.io
JavaScript2011–2013
node-js-libs / node.io

node.io

Discover this repository through its Hacker News conversations.

Web Development
Explore on GitHubRead Hacker News discussions
8Distinct people
14Mentions
0Explicit recommenders
1.8KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A public description has not been verified.

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 11:56:56 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month14 dated mentions

Observed mentions: 2011-01-22 13:51:07 UTC to 2013-08-25 23:23:38 UTC.

012342011-01-01 00:00:00 UTC to 2011-02-01 00:00:00 UTC (exclusive): 4 mentions; 1 submissions, 3 commentsJan 20112011-02-01 00:00:00 UTC to 2011-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2011-03-01 00:00:00 UTC to 2011-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2011-04-01 00:00:00 UTC to 2011-05-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2011-05-01 00:00:00 UTC to 2011-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2011-06-01 00:00:00 UTC to 2011-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2011-07-01 00:00:00 UTC to 2011-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2011-08-01 00:00:00 UTC to 2011-09-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2011-09-01 00:00:00 UTC to 2011-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2011-10-01 00:00:00 UTC to 2011-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2011-11-01 00:00:00 UTC to 2011-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2011-12-01 00:00:00 UTC to 2012-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2012-01-01 00:00:00 UTC to 2012-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2012-02-01 00:00:00 UTC to 2012-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2012-03-01 00:00:00 UTC to 2012-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2012-04-01 00:00:00 UTC to 2012-05-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsApr 20122012-05-01 00:00:00 UTC to 2012-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2012-06-01 00:00:00 UTC to 2012-07-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2012-07-01 00:00:00 UTC to 2012-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2012-08-01 00:00:00 UTC to 2012-09-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2012-09-01 00:00:00 UTC to 2012-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2012-10-01 00:00:00 UTC to 2012-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2012-11-01 00:00:00 UTC to 2012-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2012-12-01 00:00:00 UTC to 2013-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2013-01-01 00:00:00 UTC to 2013-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-02-01 00:00:00 UTC to 2013-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-03-01 00:00:00 UTC to 2013-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-04-01 00:00:00 UTC to 2013-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-05-01 00:00:00 UTC to 2013-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-06-01 00:00:00 UTC to 2013-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-07-01 00:00:00 UTC to 2013-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-08-01 00:00:00 UTC to 2013-09-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 commentsAug 2013

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Jan 20112011-01-01 00:00:00 UTC2011-02-01 00:00:00 UTC413
Feb 20112011-02-01 00:00:00 UTC2011-03-01 00:00:00 UTC000
Mar 20112011-03-01 00:00:00 UTC2011-04-01 00:00:00 UTC000
Apr 20112011-04-01 00:00:00 UTC2011-05-01 00:00:00 UTC202
May 20112011-05-01 00:00:00 UTC2011-06-01 00:00:00 UTC000
Jun 20112011-06-01 00:00:00 UTC2011-07-01 00:00:00 UTC000
Jul 20112011-07-01 00:00:00 UTC2011-08-01 00:00:00 UTC000
Aug 20112011-08-01 00:00:00 UTC2011-09-01 00:00:00 UTC101
Sep 20112011-09-01 00:00:00 UTC2011-10-01 00:00:00 UTC000
Oct 20112011-10-01 00:00:00 UTC2011-11-01 00:00:00 UTC000
Nov 20112011-11-01 00:00:00 UTC2011-12-01 00:00:00 UTC000
Dec 20112011-12-01 00:00:00 UTC2012-01-01 00:00:00 UTC000
Jan 20122012-01-01 00:00:00 UTC2012-02-01 00:00:00 UTC000
Feb 20122012-02-01 00:00:00 UTC2012-03-01 00:00:00 UTC000
Mar 20122012-03-01 00:00:00 UTC2012-04-01 00:00:00 UTC000
Apr 20122012-04-01 00:00:00 UTC2012-05-01 00:00:00 UTC101
May 20122012-05-01 00:00:00 UTC2012-06-01 00:00:00 UTC000
Jun 20122012-06-01 00:00:00 UTC2012-07-01 00:00:00 UTC101
Jul 20122012-07-01 00:00:00 UTC2012-08-01 00:00:00 UTC000
Aug 20122012-08-01 00:00:00 UTC2012-09-01 00:00:00 UTC101
Sep 20122012-09-01 00:00:00 UTC2012-10-01 00:00:00 UTC101
Oct 20122012-10-01 00:00:00 UTC2012-11-01 00:00:00 UTC000
Nov 20122012-11-01 00:00:00 UTC2012-12-01 00:00:00 UTC000
Dec 20122012-12-01 00:00:00 UTC2013-01-01 00:00:00 UTC101
Jan 20132013-01-01 00:00:00 UTC2013-02-01 00:00:00 UTC000
Feb 20132013-02-01 00:00:00 UTC2013-03-01 00:00:00 UTC000
Mar 20132013-03-01 00:00:00 UTC2013-04-01 00:00:00 UTC000
Apr 20132013-04-01 00:00:00 UTC2013-05-01 00:00:00 UTC000
May 20132013-05-01 00:00:00 UTC2013-06-01 00:00:00 UTC000
Jun 20132013-06-01 00:00:00 UTC2013-07-01 00:00:00 UTC000
Jul 20132013-07-01 00:00:00 UTC2013-08-01 00:00:00 UTC000
Aug 20132013-08-01 00:00:00 UTC2013-09-01 00:00:00 UTC202

What Hacker News says

14 sources
philipDS·Sep 28, 2012
An alternative to Python or C# for web crawling could be Node.js. It's pretty good at it and you have a few libraries that can help you: * Node.io (https://github.com/chriso/node.io) * Phantomjs-node for dynamic content (https://github.com/sgentle/phantomjs-node) * Cheerio for a jQuery server-side implementation (https://github.com/MatthewMueller/cheerio) * Node-jquery as an alternative to Cheerio (https://github.com/coolaj86/node-jquery) A single product might have a different name, but you might try to scrape the product ID's if they exist. Product IDs should be unique. If both websites provide those ID's you could compare those in your database. If that's not possible: as a small hack you could also use Amazon Mechanical Turk to issue manual tasks to compare product names. This way, real people will check if two products are the same i
Read the rest
n case there is doubt. This will cost you a little, but you could give those people 5 cents per product comparison or something like that. For question 3, some websites don't allow you to crawl their content. Read their ToS :-) For the rest I agree with Piskvorrr, you could do some trial and error and learn on the fly or read some books (and still do trial and error and learn on the fly ;-)). Good luck!
In Ask HN: How to create a Web crawler?
chrisohara·Aug 5, 2011
I get simple scalability, server-client code reuse, easy realtime, and, despite what you or others may think, I like Javascript as a language. Node.JS was also a perfect fit for the scraping framework I wrote: https://github.com/chriso/node.io At one stage I had the same train of thought that you have (http://chris6f.com/synchronous-nodejs / https://github.com/chriso/synchronous/blob/master/lib/protot...) - it would be nice to have the option of fibers, but it's not going to happen. IMO async code isn't as difficult or ugly as you make it out to be. Is async code as easy to write and follow as sync code? No. Is it worth the benefits I've mentioned? For me, yes.
In Node.js - A Giant Step Backwards
chrisohara·Apr 15, 2011
Node.JS seemed like a perfect fit for a few reason: 1. JS selectors make scraping _very_ easy. 2. Asynchronous is fast as it is, but the page is actually parsed as it's received - contrast this with other scraping solutions where you need to download a page and parse it once it's complete. 3. With asynchronous scraping it's trivial to handle failures, timeouts, retries, nested requests, recursing similar URLs, concurrent requests, etc. - just add one of the many options (https://github.com/chriso/node.io/wiki/API---Job-Options)
In Scraping the web with Node.io
simonw·Jan 23, 2011
I was intrigued to see what CSS selector engine it was using... https://github.com/chriso/node.io uses https://github.com/harryf/node-soupselect https://github.com/harryf/node-soupselect is a port of my https://github.com/simonw/soupselect library for Python https://github.com/simonw/soupselect is a port of my getElementsBySelector function for JavaScript: http://simonwillison.net/2003/Mar/25/getElementsBySelector/ I'm always surprised to see that code still being used - it's the least complete selector library out there by a long way.
In Pro scraping with Node.JS

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.