JAjamesmurdza·Mar 20, 2024
Nice. I've used Cheerio for this in the past: https://github.com/cheeriojs/cheerio?tab=readme-ov-file#sele...
In Htmlq: Like Jq, but for HTMLcheeriojs
cheerio
TypeScript2011–2024
cheeriojs / cheerio
cheerio
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
14Distinct people
15Mentions
1Explicit recommenders
30.5KGitHub stars
What to know before exploring
What it is
The fast, flexible, and elegant library for parsing and manipulating HTML and XML.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
Hmm, I can see why I'd want to use this library over piecing together requests and BS4 myself for every project. I love how simple the examples look. I have a project I'm…
pcr0 used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year15 dated mentions
Observed mentions: 2011-11-10 05:12:42 UTC to 2024-03-20 00:12:48 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2011 | 2011-01-01 00:00:00 UTC | 2012-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2012 | 2012-01-01 00:00:00 UTC | 2013-01-01 00:00:00 UTC | 4 | 1 | 3 |
| 2013 | 2013-01-01 00:00:00 UTC | 2014-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2014 | 2014-01-01 00:00:00 UTC | 2015-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 4 | 0 | 4 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 2 | 0 | 2 |
What Hacker News says
15 sources> I'm not sure why Python web scraping is so popular compared to Node.js web scraping
Take this with a grain of salt, since I am fully cognizant that I'm the outlier in most of these conversations, but Scrapy is A++ the no-kidding best framework for this activity that has been created thus far. So, if there was scrapyjs maybe I'd look into it, but there's not (that I'm aware of) so here we are. This conversation often comes up in any such "well, I just use requests & ..." conversation and if one is happy with main.py and a bunch of requests invocations, I'm glad for you, but I don't want to try and cobble together all the side-band stuff that Scrapy and its ecosystem provide for me in a reusable and predictable way
Also, often those conversations conflate the server side language with the "scrape using headed browser" language which happe
Read the rest
ns to be the same one. So, if one is using cheerio <https://github.com/cheeriojs/cheerio> then sure node can be a fine thing - if the blog post is all "fire up puppeteer, what can go wrong?!" then there is the road to ruin of doing battle with all kinds of detection problems since it's kind of a browser but kind of not
I, under no circumstances, want the target site running their JS during my crawl runs. I fully accept responsibility for reproducing any XHR or auth or whatever to find the 3 URLs that I care about, without downloading every thumbnail and marketing JS and beacon and and and. I'm also cognizant that my traffic will thus stand out since it uniquely does not make the beacon and marketing calls, but my experience has been that I get the ban hammer less often with my target fetches than trying to pretend to be a browser with a human on the keyboard/mouse but is not
Sure, it's actually been a long journey since PHP.
I switched from PHP to Node about 11 years ago. I was probably one of the first 100 Node.js developers. My biggest contribution there was creating https://github.com/cheeriojs/cheerio.
Go spoils me because it "just works". The best way I can explain it is when I'm stuck on something in Go, it's almost always my bad, not something that Go is doing poorly. This is in contrast to the Node.js ecosystem where it feels like half of the problems I run into shouldn't be my responsibility.
I strongly relate to this post by Kevin Burke: https://kevin.burke.dev/kevin/one-year-of-node-js/. A lot has improved since this was written. We're finally over the ES Module vs CommonJS hump, Typescript has become the defacto standard, Babel is less used now and Webpack is effectively abstracted away. I still r
Read the rest
un into way too many problems with the ecosystem and the transition to Rust will make the ecosystem more complicated, not less.
My transition from Node.js to Go happened because I was working solo on https://standupjack.com/ and I kept running into issues with the message broker. It was using the most popular RabbitMQ client for Node, yet it was still super flaky.
About 6 years ago, I was chatting with TJ Holowaychuk and he kept singing Go's praises. One holiday I decided to rewrite just the message broker in Go. I got it into production in 3 days and didn't have anymore problems with it. That set me on the Go journey :)
With node, you can use cheerio [0]. It allows you to parse html pages with a JQuery similar syntax. I use it in production on my project [1]
[0] https://github.com/cheeriojs/cheerio
[1] https://github.com/Softcadbury/football-peek/blob/master/ser...
In Ask HN: What are best tools for web scraping?Check out https://github.com/cheeriojs/cheerio
In Ask HN: Web Scraping with Node.jsAny language with support for web crawling? I can sorta see it being done in Node with cheerio[1] and an HTTP client.
[1] https://github.com/cheeriojs/cheerio
In An industry that helps Chinese cheat their way into and through US collegesHmm, I can see why I'd want to use this library over piecing together requests and BS4 myself for every project. I love how simple the examples look.
I have a project I'm working on that will involve scraping many different websites on a daily basis. My only scraping experience so far is using cheerio[0] to scrape a single page with a 1,000 row HTML table. Should I start with something BS-based like this or should I jump straight into Scrapy? Or are there any other alternatives I should try?
[0]: https://github.com/cheeriojs/cheerio
In RoboBrowser: Your friendly neighborhood web scraperHeh. I still use jQuery for pretty much every project.
There is a server-side port of jQuery which is absolutely fantastic: https://github.com/cheeriojs/cheerio This allows you to render jQuery based views on the server or client with the same code.
All the latest JavaScript view libraries are a huge pain to work with. Sharp learning curves, punishing APIs, and extremely unexpected performance issues. jQuery just works.
<3 jQuery
In A love letter to jQueryI use custom node.js scripts with these libraries:
* request - https://github.com/mikeal/request
* async - https://github.com/caolan/async
* cheerio - https://github.com/cheeriojs/cheerio
* nedb - https://github.com/louischatriot/nedb
In Import.io – Structured Web Data ScrapingDid you bother to read the post? It pitted two different DOM traversal libraries against each other:
* cheerio (https://github.com/MatthewMueller/cheerio)
* PhpQuery (https://code.google.com/p/phpquery/wiki/jQueryPortingState)
Both of these use a jQuery-esque syntax, so your comment regarding DOM traversal in PHP is a moot point.
In Benchmarking Codswallop: Node.js vs. PHPBetter than that is http://node.io/ Also, don't use jsdom (it is slow and strict), https://github.com/MatthewMueller/cheerio is much better.
In I Don’t Need No Stinking API: Web Scraping For Fun and ProfitAn alternative to Python or C# for web crawling could be Node.js. It's pretty good at it and you have a few libraries that can help you:
* Node.io (https://github.com/chriso/node.io)
* Phantomjs-node for dynamic content (https://github.com/sgentle/phantomjs-node)
* Cheerio for a jQuery server-side implementation (https://github.com/MatthewMueller/cheerio)
* Node-jquery as an alternative to Cheerio (https://github.com/coolaj86/node-jquery)
A single product might have a different name, but you might try to scrape the product ID's if they exist. Product IDs should be unique. If both websites provide those ID's you could compare those in your database.
If that's not possible: as a small hack you could also use Amazon Mechanical Turk to issue manual tasks to compare product names. This way, real people will check if two products are the same i
Read the rest
n case there is doubt. This will cost you a little, but you could give those people 5 cents per product comparison or something like that.
For question 3, some websites don't allow you to crawl their content. Read their ToS :-)
For the rest I agree with Piskvorrr, you could do some trial and error and learn on the fly or read some books (and still do trial and error and learn on the fly ;-)). Good luck!
I have also found node+jQuery an effective web crawling combination. In particular the cheerio library https://github.com/MatthewMueller/cheerio greatly simplifies data extraction. And as others have mentioned, the asynchronous nature of node is perfectly suited to crawling (as long as you take care not to accidentally DDOS the target site).
In Using Node.js and JQuery to Crawl Public TweetsCheerio - fast, flexible, and lean implementation of core jQuery for the node.js
5 points · Open discussion