HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
apify
crawlee
TypeScript2018–2025
apify / crawlee

crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Web DevelopmentAI & Machine LearningProductivityNetworkingAutomation & WorkflowsLibraries & SDKs
Explore on GitHubRead Hacker News discussions
8Distinct people
10Mentions
0Explicit recommenders
26.4KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year10 dated mentions

Observed mentions: 2018-06-02 03:19:56 UTC to 2025-10-03 10:00:28 UTC.

01232018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments20182019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 3 mentions; 2 submissions, 1 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC312
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC000
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC000
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC000
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC303
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC321
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC101

What Hacker News says

10 sources
renegat0x0·Oct 3, 2025
What I have seen it is hard to tell what "serious scrapers" use. They use many things. Some use this, some not. This is what I have learned reading webscraping on reddit. Nobody speaks things like that out loud. There are many tools, see links below Personally I think that running selenium can be a bottle neck, as it does not play nice, sometimes processes break, even system sometimes requires restart because of things blocked, can be memory hog, etc. etc. That is my experience. To be able to scale I think you have to have your own implementation. Serious scrapers complain about people using selenium, or derivatives as noobs, who will come back asking why page X does not work in scraping mechanisms. https://github.com/lexiforest/curl_cffi https://github.com/encode/httpx https://github.com/scrapy/scrapy https://github.com/apify/crawlee
In How I block all 26M of your curl requests
jancurn·Jul 9, 2024
The main advantage (for now) is that the library has a single interface for both HTTP and headless browsers, and bundled auto scaling. You can write your crawlers using the same base abstraction, and the framework takes care of this heavy lifting. Developers of scrapers shouldn't need to reinvent the wheel, and just focus on building the "business" logic of their scrapers. Having said that, if you wrote your own crawling library, the motivation to use Crawlee might be lower, and that's fair enough. Please note that this is the first release, and we'll keep adding many more features as we go, including anti-blocking, adaptive crawling, etc. To see where this might go, check https://github.com/apify/crawlee
In Show HN: Crawlee for Python – a web scraping and browser automation library
jancurn·Jul 9, 2024Submission

Show HN: Crawlee for Python – a web scraping and browser automation library

Hey all, This is Jan, the founder of Apify (https://apify.com/) — a full-stack web scraping platform. After the success of Crawlee for JavaScript (https://github.com/apify/crawlee/) and the demand from the Python community, we're launching Crawlee for Python today! The main features are: - A unified programming interface for both HTTP (HTTPX with BeautifulSoup) & headless browser crawling (Playwright) - Automatic parallel crawling based on available system resources - Written in Python with type hints for enhanced developer experience - Automatic retries on errors or when you’re getting blocked - Integrated proxy rotation and session management - Configurable request routing - direct URLs to the appropriate handlers - Persistent queue for URLs to crawl - Pluggable storage for both tabular data and files For details, you can read the annou
Read the rest
ncement blog post: https://crawlee.dev/blog/launching-crawlee-python Our team and I will be happy to answer here any questions you might have.
254 points · Open discussion
jonatron·Aug 23, 2022
I'm not aware of a benchmark, but puppeteer-extra-plugin-stealth can be detected: https://datadome.co/bot-management-protection/detecting-head... Crawlee does appear to do the basic checks though, like checking navigator.webdriver: https://github.com/apify/crawlee/blob/master/test/browser-po... Last time I checked (over a year ago) I couldn't find any public code to make Chrome/Firefox properly undetectable. That said, going to extreme lengths to be undetectable is rarely necessary, because some sites will serve up CAPTCHA's to real people on clean uncompromised residential connections anyway.
In Show HN: Crawlee – Web scraping and browser automation library for Node.js
nlh·Aug 10, 2022
I'm working on a personal project that involves A LOT of scraping, and through several iterations I've gotten some stuff that works quite well. Here's a quick summary of what I've explored (both paid and free): * Apify (https://apify.com/) is a great, comprehensive system if you need to get fairly low-level. Everything is hosted there, they've got their own proxy service (or you can roll your own), and their open source framework (https://github.com/apify/crawlee) is excellent. * I've also experimented with running both their SDK (crawlee) and Playwright directly on Google Cloud Run, and that also works well and is an order-of-magnitude less expensive than running directly on their platform. * Bright Data nee Luminati is excellent for cheap data center proxies ($0.65/GB pay as you go), but prices get several orders of magnitude more exp
Read the rest
ensive if you need anything more thorough than data center proxies. * For some direct API crawls that I do, all of the scraping stuff is unnecessary and I just ping the APIs directly. * If the site you're scraping is using any sort of anti-bot protection, I've found that ScrapingBee (https://www.scrapingbee.com/) is by far the easiest solution. I spent many many hours fighting anti-bot protection doing it myself with some combination of Bright Data, Apify and Playwright, and in the end I kinda stopped battling and just decided to let ScrapingBee deal with it for me. I may be lucky in that the sites I'm scraping don't really use JS heavily, so the plain vanilla, no-JS ScrapingBee service works almost all of the time for those. Otherwise it can get quite expensive if you need JS rendering, premium proxies, etc. But a big thumbs up to them for making it really easy. Always looking for new techniques and tools, so I'll monitor this thread closely.
In Ask HN: What are the best tools for web scraping in 2022?
mtrunkat·Jun 5, 2018
Problem with fixed concurrency is that required memory per Chrome process varies a lot. We (https://www.apify.com) are solving this by autoscaling number of parallel Puppeteer instances based on memory and CPU. Our open source SDK (http://github.com/apifytech/apify-js) implements this using class PuppeteerCrawler (https://www.apify.com/docs/sdk/apify-runtime-js/latest#Puppe...) which internally uses AutoscaledPool that provides autoscaling: - https://github.com/apifytech/apify-js/blob/master/src/autosc... - https://www.apify.com/docs/sdk/apify-runtime-js/latest#Autos... Sadly this feature is currently limited to our platform because it's mainly build for running in Docker containers. And as Docker container don't know about it's CPU consumption vs limits it requires a notifications about reaching its CPU limit from underlying platform. We
Read the rest
have solved this using Websocket events. But we are currently working on extending this to work anywhere/locally.
In Observations running 2M headless sessions
eboyjr·Jun 2, 2018
> we have developed a solution which removes the property from the web browser and thus prevents these kind of protections from figuring out that the browser is automated Eli Grey and I have bypassed your "hideWebDriver()" function[1] in a single line of code: if (navigator.webdriver || (Navigator.prototype && Object.getOwnPropertyDescriptors(Navigator.prototype)["webdriver"])) { // Chrome headless detected - navigator.webdriver exists or was redefined } [1]: https://github.com/apifytech/apify-js/blob/262a2e604b1adb3d8...
In Bypassing website anti-scraping protections

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.