CAcavisne·Mar 18, 2025
In AI crawlers haven't learned to play nice with websitesgoogle
robotstxt
C++2019–2025
google / robotstxt
robotstxt
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
7Distinct people
8Mentions
0Explicit recommenders
3.5KGitHub stars
What to know before exploring
What it is
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year8 dated mentions
Observed mentions: 2019-07-01 14:26:05 UTC to 2025-03-18 14:43:50 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 7 | 0 | 7 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
8 sourcesAdditionally: do you allow for words to be spelled incorrectly? People fat-finger things all the time, more so on mobile devices. Many pieces of software have this to some degree, see robots.txt as an example:
https://github.com/google/robotstxt/blob/bacd8ba94b6d066ff6c...
In Rules for Intuitive UXI absolutely understand why they did this, but I have to say I was disappointed to see only 7 commits at https://github.com/google/robotstxt/commits/master dating back to June 25th.
When I read "This library has been around for 20 years and it contains pieces of code that were written in the 90's" my first thought was "that commit history must be FASCINATING".
In Google’s robots.txt parser is now open sourceI guess lots of people misspell ~disalow~ disallow[1]
1. https://github.com/google/robotstxt/blob/master/robots.cc#L6...
In Google’s robots.txt parser is now open sourceGoogle is also really generous with how they will let you spell "disallow": https://github.com/google/robotstxt/blob/59f3643d3a3ac88f613...
:D
In Google’s robots.txt parser is now open sourceI've been in disagreements with SEO people quite frequently about a "Noindex" directive for robots.txt. There seem to be a bunch of articles that are sent to me every time I question its existence[0][1]. Google's own documentation says that noindex should be in the meta HTML but the SEO people seem to trust these shady sites more.
I haven't read through all of the code but it assuming this is actually what's running on Google's scrapers this section [2] seems to be pretty conclusive evidence to me that this Noindex thing is bullshit.
[0] https://www.deepcrawl.com/blog/best-practice/robots-txt-noin...
[1]https://www.stonetemple.com/does-google-respect-robots-txt-n...
[2] https://github.com/google/robotstxt/blob/59f3643d3a3ac88f613...
In Google’s robots.txt parser is now open sourceNote that this is quite strict on what characters may be contained in a bots user agent. This is due to strictness in the REP standard.
https://github.com/google/robotstxt/blob/master/robots_test....
// A user-agent line is expected to contain only [a-zA-Z_-] characters and must
// not be empty. See REP I-D section "The user-agent line".
// https://tools.ietf.org/html/draft-rep-wg-topic#section-2.2.1
So you may need to adjust your bot’s UA for proper matching.
(Disclosure, I work at Google, though not on anything related to this.)
In Google’s robots.txt parser is now open sourceCode here: https://github.com/google/robotstxt
In Google’s robots.txt parser is now open source