HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
scrapy
scrapyd
Python2020–2022
scrapy / scrapyd

scrapyd

A service daemon to run Scrapy spiders

Explore on GitHubRead Hacker News discussions
2Distinct people
2Mentions
0Explicit recommenders
3.1KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A service daemon to run Scrapy spiders

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month2 dated mentions

Observed mentions: 2020-07-21 19:50:06 UTC to 2022-08-10 15:32:19 UTC.

012020-07-01 00:00:00 UTC to 2020-08-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsJul 20202020-08-01 00:00:00 UTC to 2020-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-09-01 00:00:00 UTC to 2020-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-10-01 00:00:00 UTC to 2020-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-11-01 00:00:00 UTC to 2020-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-12-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-01 00:00:00 UTC to 2021-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-02-01 00:00:00 UTC to 2021-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-03-01 00:00:00 UTC to 2021-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-04-01 00:00:00 UTC to 2021-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-05-01 00:00:00 UTC to 2021-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-06-01 00:00:00 UTC to 2021-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-07-01 00:00:00 UTC to 2021-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsJul 20212021-08-01 00:00:00 UTC to 2021-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-09-01 00:00:00 UTC to 2021-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-10-01 00:00:00 UTC to 2021-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-11-01 00:00:00 UTC to 2021-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-12-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-01-01 00:00:00 UTC to 2022-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-02-01 00:00:00 UTC to 2022-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-03-01 00:00:00 UTC to 2022-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-04-01 00:00:00 UTC to 2022-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-05-01 00:00:00 UTC to 2022-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-06-01 00:00:00 UTC to 2022-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-07-01 00:00:00 UTC to 2022-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-08-01 00:00:00 UTC to 2022-09-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsAug 2022

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Jul 20202020-07-01 00:00:00 UTC2020-08-01 00:00:00 UTC101
Aug 20202020-08-01 00:00:00 UTC2020-09-01 00:00:00 UTC000
Sep 20202020-09-01 00:00:00 UTC2020-10-01 00:00:00 UTC000
Oct 20202020-10-01 00:00:00 UTC2020-11-01 00:00:00 UTC000
Nov 20202020-11-01 00:00:00 UTC2020-12-01 00:00:00 UTC000
Dec 20202020-12-01 00:00:00 UTC2021-01-01 00:00:00 UTC000
Jan 20212021-01-01 00:00:00 UTC2021-02-01 00:00:00 UTC000
Feb 20212021-02-01 00:00:00 UTC2021-03-01 00:00:00 UTC000
Mar 20212021-03-01 00:00:00 UTC2021-04-01 00:00:00 UTC000
Apr 20212021-04-01 00:00:00 UTC2021-05-01 00:00:00 UTC000
May 20212021-05-01 00:00:00 UTC2021-06-01 00:00:00 UTC000
Jun 20212021-06-01 00:00:00 UTC2021-07-01 00:00:00 UTC000
Jul 20212021-07-01 00:00:00 UTC2021-08-01 00:00:00 UTC000
Aug 20212021-08-01 00:00:00 UTC2021-09-01 00:00:00 UTC000
Sep 20212021-09-01 00:00:00 UTC2021-10-01 00:00:00 UTC000
Oct 20212021-10-01 00:00:00 UTC2021-11-01 00:00:00 UTC000
Nov 20212021-11-01 00:00:00 UTC2021-12-01 00:00:00 UTC000
Dec 20212021-12-01 00:00:00 UTC2022-01-01 00:00:00 UTC000
Jan 20222022-01-01 00:00:00 UTC2022-02-01 00:00:00 UTC000
Feb 20222022-02-01 00:00:00 UTC2022-03-01 00:00:00 UTC000
Mar 20222022-03-01 00:00:00 UTC2022-04-01 00:00:00 UTC000
Apr 20222022-04-01 00:00:00 UTC2022-05-01 00:00:00 UTC000
May 20222022-05-01 00:00:00 UTC2022-06-01 00:00:00 UTC000
Jun 20222022-06-01 00:00:00 UTC2022-07-01 00:00:00 UTC000
Jul 20222022-07-01 00:00:00 UTC2022-08-01 00:00:00 UTC000
Aug 20222022-08-01 00:00:00 UTC2022-09-01 00:00:00 UTC101

What Hacker News says

2 sources
Stefan_Smirnov·Aug 10, 2022
I’m biased since I’m an owner of a web scraping agency (https://webscrapingsolutions.co.uk/). I was asking myself the same question in 2019. You can use any programming language, but have settled on this tech-stack Python, Scrapy (https://github.com/scrapy/scrapy), Redis, PostgreSQL. for the following reasons: [1] Scrapy is a well-documented framework, so any Python programmer can start using it after 1 month of training. There are a lot of guides for beginners. [2] Lots of features are already implemented and open-source, you won’t have to waste time & money on them. [3] There is a strong community that can help with most of the questions (I don't think any other alternative has that). [4] Scrapy developers are cheap. You will only need junior+ to middle level software engineers to pull out most of the projects. It’s not rocket since. [5
Read the rest
] Recruiting is easier: - there are hundreds of freelancers with relevant expertise - if you search on LinkedIn - there are hundreds of software developers that have worked with Scrapy in the past, and you don’t need that many - you can grow expertise in your own team quickly - developers are easily replaceable, even on larger projects - you can use the same developers on backend tasks. [6] You don’t need a DevOps expertise in your web scraping team because Scrapy Cloud (https://www.zyte.com/scrapy-cloud/) is good and cheap enough for 99% of the projects. [7] If you decide to have your own infrastructure, you can use https://github.com/scrapy/scrapyd. [8] The entire ecosystem is well-well-maintained and steadily growing. You can integrate a lot of 3-rd party services into your project within hours: proxies, captcha solving, headless browsers, HTML parsing APIs. [9] It’s easy to integrate your own AI/ML models into the scraping workflow. [10]. With some work, you can use Scrapy for distributed projects that are scraping thousands (millions) of domains. We are using https://github.com/rmax/scrapy-redis. [11] Commercial support is available. There are several companies that can develop you an entire project or take over an existing one - if you don’t have the time/don’t want to do it on your own. We have built dozens of projects in multiple industries: - news monitoring - job aggregators - real estate aggregators - ecommerce (anything from 1 website, to monitoring prices on 100k+ domains) - lead generation - search engines in a specific niche (SEO, pdf files, ecommerce, chemical retail) - macroeconomic research & indicators - social media, NFT marketplaces, etc So, most of the projects can be finished using these tools.
In Ask HN: What are the best tools for web scraping in 2022?
mdaniel·Jul 21, 2020
what does the Realtime mean? I believe they are contrasting it with how something like https://github.com/scrapy/scrapyd#readme or the ubiquitous "run scrapy under crontab" setups normally function: you kick off the spider, then you go get coffee and wait for the ItemPipeline to deliver the results somewhere else With scrapyrt, you submit a POST or GET, and that URL blocks until the one requested thing completes or times out, enabling (only conceptually, mind you) an archive.is replacement using an XHR
In Scrapinghub/Scrapyrt: Scrapy Realtime

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.