HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
scrapy
scrapy
Python2011–2025
scrapy / scrapy

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

Web DevelopmentAutomation & Workflows
Explore on GitHubRead Hacker News discussions
17Distinct people
19Mentions
0Explicit recommenders
65.4KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Scrapy, a fast high-level web crawling & scraping framework for Python.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year19 dated mentions

Observed mentions: 2011-11-17 18:26:29 UTC to 2025-10-03 10:00:28 UTC.

0123452011-01-01 00:00:00 UTC to 2012-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments20112012-01-01 00:00:00 UTC to 2013-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2013-01-01 00:00:00 UTC to 2014-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 5 mentions; 0 submissions, 5 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments20182019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20112011-01-01 00:00:00 UTC2012-01-01 00:00:00 UTC110
20122012-01-01 00:00:00 UTC2013-01-01 00:00:00 UTC000
20132013-01-01 00:00:00 UTC2014-01-01 00:00:00 UTC000
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC303
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC202
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC505
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC202
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC000
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC000
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC000
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC101
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC312
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC211

What Hacker News says

19 sources
renegat0x0·Oct 3, 2025
What I have seen it is hard to tell what "serious scrapers" use. They use many things. Some use this, some not. This is what I have learned reading webscraping on reddit. Nobody speaks things like that out loud. There are many tools, see links below Personally I think that running selenium can be a bottle neck, as it does not play nice, sometimes processes break, even system sometimes requires restart because of things blocked, can be memory hog, etc. etc. That is my experience. To be able to scale I think you have to have your own implementation. Serious scrapers complain about people using selenium, or derivatives as noobs, who will come back asking why page X does not work in scraping mechanisms. https://github.com/lexiforest/curl_cffi https://github.com/encode/httpx https://github.com/scrapy/scrapy https://github.com/apify/crawlee
In How I block all 26M of your curl requests
mdaniel·Nov 11, 2022
If you weren't already aware, Scrapy has strong support for this via their HTTPCache middleware; you can choose whether to have it actually behave like a cache, choosing to returned already scraped content if matched or merely to act as a pass-through cache: https://docs.scrapy.org/en/2.7/topics/downloader-middleware.... Their OOtB storage does what the sibling comment says about sha1-ing the request and then sharding the output filename by the first two characters: https://github.com/scrapy/scrapy/blob/2.7.1/scrapy/extension...
In Ask HN: Best way to keep the raw HTML of scraped pages?
Stefan_Smirnov·Aug 10, 2022
I’m biased since I’m an owner of a web scraping agency (https://webscrapingsolutions.co.uk/). I was asking myself the same question in 2019. You can use any programming language, but have settled on this tech-stack Python, Scrapy (https://github.com/scrapy/scrapy), Redis, PostgreSQL. for the following reasons: [1] Scrapy is a well-documented framework, so any Python programmer can start using it after 1 month of training. There are a lot of guides for beginners. [2] Lots of features are already implemented and open-source, you won’t have to waste time & money on them. [3] There is a strong community that can help with most of the questions (I don't think any other alternative has that). [4] Scrapy developers are cheap. You will only need junior+ to middle level software engineers to pull out most of the projects. It’s not rocket since. [5
Read the rest
] Recruiting is easier: - there are hundreds of freelancers with relevant expertise - if you search on LinkedIn - there are hundreds of software developers that have worked with Scrapy in the past, and you don’t need that many - you can grow expertise in your own team quickly - developers are easily replaceable, even on larger projects - you can use the same developers on backend tasks. [6] You don’t need a DevOps expertise in your web scraping team because Scrapy Cloud (https://www.zyte.com/scrapy-cloud/) is good and cheap enough for 99% of the projects. [7] If you decide to have your own infrastructure, you can use https://github.com/scrapy/scrapyd. [8] The entire ecosystem is well-well-maintained and steadily growing. You can integrate a lot of 3-rd party services into your project within hours: proxies, captcha solving, headless browsers, HTML parsing APIs. [9] It’s easy to integrate your own AI/ML models into the scraping workflow. [10]. With some work, you can use Scrapy for distributed projects that are scraping thousands (millions) of domains. We are using https://github.com/rmax/scrapy-redis. [11] Commercial support is available. There are several companies that can develop you an entire project or take over an existing one - if you don’t have the time/don’t want to do it on your own. We have built dozens of projects in multiple industries: - news monitoring - job aggregators - real estate aggregators - ecommerce (anything from 1 website, to monitoring prices on 100k+ domains) - lead generation - search engines in a specific niche (SEO, pdf files, ecommerce, chemical retail) - macroeconomic research & indicators - social media, NFT marketplaces, etc So, most of the projects can be finished using these tools.
In Ask HN: What are the best tools for web scraping in 2022?
mnmkng·Sep 14, 2021
While I agree that Scrapy is a great tool for beginner tutorials and easy entry into scraping, it's becoming difficult to use it in real world scenarios because almost all the large players now employ some anti-bot or anti-scraping protection. A great example above all is Cloudflare. You simply can't convince Cloudflare you're a human with Scrapy alone. Scrapy has only experimental support of HTTP2 and does not support proxies over HTTP2 (https://github.com/scrapy/scrapy/issues/5213). Yet, all browsers use HTTP2 now, which means all normal users use HTTP2... You get the point. What we use now is Got Scraping (https://github.com/apify/got-scraping). It's a special purpose extension of Got (HTTP client with 18 mil weekly downloads) that masks its HTTP communication as if it was coming from a real browser. Of course, this will not get you as
Read the rest
far as Puppeteer or Playwright (headless browsers), but it improved our scraping tremendously. If you need a full crawling library, see the Apify SDK (https://sdk.apify.com) which uses Got Scraping under the hood.
In How to Crawl the Web with Scrapy
kmike84·Mar 21, 2016
I was investigating how to add asyncio / async def support to Scrapy (see https://github.com/scrapy/scrapy/issues/1144#issuecomment-14...). Small examples like the one at the link look neat, but it is not all roses as you go further. The problems are not specific to Scrapy; I think any advanced `async def` based crawler will face them. There are 2 challenges with async def I don't know how to solve elegantly: 1. how to integrate coroutine-based scraping code with on-disk persistent request queues; 2. how to deallocate resources without boilerplate in coroutine-based scraping code. (1) is easier with callbacks-as-methods because this way state is passed explicitly (it is not in local variables), so Scrapy can choose to save it to disk. Example of (2) is this code: async def parse(self, response): resp = await self.fetch(url)
Read the rest
# ... find another URL to follow # Here we have the problem: # response object is kept in memory # until second response is fully received. # This is a problem if 10s and 100s # of requests are processed in parallel # and responses are large. # Because of refcounting, with callbacks # response would have been kept in # memory only until second request # starts - callbacks+refcounting provide # an elegant way for resource deallocation. resp = await self.fetch(url2) If anyone has suggestions please comment on https://github.com/scrapy/scrapy/issues/1144#issuecomment-14....
In Show HN: Python 3.5 Async Web Crawler Example

Showing the 15 most recent of 19 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.