HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
rmax
scrapy-redis
Python2014–2022
rmax / scrapy-redis

scrapy-redis

Redis-based components for Scrapy.

Data & DatabasesSystems & InfrastructureAutomation & Workflows
Explore on GitHubRead Hacker News discussions
4Distinct people
4Mentions
0Explicit recommenders
5.6KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Redis-based components for Scrapy.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 9:01:12 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year4 dated mentions

Observed mentions: 2014-03-11 03:36:32 UTC to 2022-08-10 15:32:19 UTC.

0122014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments20142015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments20182019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2022

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC202
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC000
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC000
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC101
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC000
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC000
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC000
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC000
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC101

What Hacker News says

1 sources
Stefan_Smirnov·Aug 10, 2022
I’m biased since I’m an owner of a web scraping agency (https://webscrapingsolutions.co.uk/). I was asking myself the same question in 2019. You can use any programming language, but have settled on this tech-stack Python, Scrapy (https://github.com/scrapy/scrapy), Redis, PostgreSQL. for the following reasons: [1] Scrapy is a well-documented framework, so any Python programmer can start using it after 1 month of training. There are a lot of guides for beginners. [2] Lots of features are already implemented and open-source, you won’t have to waste time & money on them. [3] There is a strong community that can help with most of the questions (I don't think any other alternative has that). [4] Scrapy developers are cheap. You will only need junior+ to middle level software engineers to pull out most of the projects. It’s not rocket since. [5
Read the rest
] Recruiting is easier: - there are hundreds of freelancers with relevant expertise - if you search on LinkedIn - there are hundreds of software developers that have worked with Scrapy in the past, and you don’t need that many - you can grow expertise in your own team quickly - developers are easily replaceable, even on larger projects - you can use the same developers on backend tasks. [6] You don’t need a DevOps expertise in your web scraping team because Scrapy Cloud (https://www.zyte.com/scrapy-cloud/) is good and cheap enough for 99% of the projects. [7] If you decide to have your own infrastructure, you can use https://github.com/scrapy/scrapyd. [8] The entire ecosystem is well-well-maintained and steadily growing. You can integrate a lot of 3-rd party services into your project within hours: proxies, captcha solving, headless browsers, HTML parsing APIs. [9] It’s easy to integrate your own AI/ML models into the scraping workflow. [10]. With some work, you can use Scrapy for distributed projects that are scraping thousands (millions) of domains. We are using https://github.com/rmax/scrapy-redis. [11] Commercial support is available. There are several companies that can develop you an entire project or take over an existing one - if you don’t have the time/don’t want to do it on your own. We have built dozens of projects in multiple industries: - news monitoring - job aggregators - real estate aggregators - ecommerce (anything from 1 website, to monitoring prices on 100k+ domains) - lead generation - search engines in a specific niche (SEO, pdf files, ecommerce, chemical retail) - macroeconomic research & indicators - social media, NFT marketplaces, etc So, most of the projects can be finished using these tools.
In Ask HN: What are the best tools for web scraping in 2022?

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.