DMdmit·Feb 11, 2026
In End of an era for me: no more self-hosted gitai-robots-txt
ai.robots.txt
Python2024–2026
ai-robots-txt / ai.robots.txt
ai.robots.txt
A list of AI agents and robots to block.
17Distinct people
22Mentions
0Explicit recommenders
4.3KGitHub stars
What to know before exploring
What it is
A list of AI agents and robots to block.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month22 dated mentions
Observed mentions: 2024-03-28 11:46:09 UTC to 2026-02-11 22:48:21 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Mar 2024 | 2024-03-01 00:00:00 UTC | 2024-04-01 00:00:00 UTC | 1 | 1 | 0 |
| Apr 2024 | 2024-04-01 00:00:00 UTC | 2024-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2024 | 2024-05-01 00:00:00 UTC | 2024-06-01 00:00:00 UTC | 0 | 0 | 0 |
| Jun 2024 | 2024-06-01 00:00:00 UTC | 2024-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2024 | 2024-07-01 00:00:00 UTC | 2024-08-01 00:00:00 UTC | 2 | 0 | 2 |
| Aug 2024 | 2024-08-01 00:00:00 UTC | 2024-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2024 | 2024-09-01 00:00:00 UTC | 2024-10-01 00:00:00 UTC | 3 | 1 | 2 |
| Oct 2024 | 2024-10-01 00:00:00 UTC | 2024-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2024 | 2024-11-01 00:00:00 UTC | 2024-12-01 00:00:00 UTC | 1 | 0 | 1 |
| Dec 2024 | 2024-12-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 3 | 0 | 3 |
| Jan 2025 | 2025-01-01 00:00:00 UTC | 2025-02-01 00:00:00 UTC | 5 | 0 | 5 |
| Feb 2025 | 2025-02-01 00:00:00 UTC | 2025-03-01 00:00:00 UTC | 1 | 0 | 1 |
| Mar 2025 | 2025-03-01 00:00:00 UTC | 2025-04-01 00:00:00 UTC | 1 | 0 | 1 |
| Apr 2025 | 2025-04-01 00:00:00 UTC | 2025-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2025 | 2025-05-01 00:00:00 UTC | 2025-06-01 00:00:00 UTC | 1 | 0 | 1 |
| Jun 2025 | 2025-06-01 00:00:00 UTC | 2025-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2025 | 2025-07-01 00:00:00 UTC | 2025-08-01 00:00:00 UTC | 1 | 0 | 1 |
| Aug 2025 | 2025-08-01 00:00:00 UTC | 2025-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2025 | 2025-09-01 00:00:00 UTC | 2025-10-01 00:00:00 UTC | 1 | 1 | 0 |
| Oct 2025 | 2025-10-01 00:00:00 UTC | 2025-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2025 | 2025-11-01 00:00:00 UTC | 2025-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2025 | 2025-12-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 1 | 0 | 1 |
| Jan 2026 | 2026-01-01 00:00:00 UTC | 2026-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2026 | 2026-02-01 00:00:00 UTC | 2026-03-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
22 sourcesUser agents not IPs, but: https://github.com/ai-robots-txt/ai.robots.txt
In Stop crawling my HTML – use the APIai.robots.txt – A list of AI agents and robots to block
2 points · Open discussionHey OP,
1)
You consider this about the Linkedin site but don't stop to think about other social networks. This is true about basically all of them. You may not post on Facebook, Bluesky, etc, but other people may like your links and post them there.
I recently ran into this as it turns out the Facebook entries in https://github.com/ai-robots-txt/ai.robots.txt also block the crawler FB uses for link previews.
2)
From your first post,
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that.
Kinda .... it's technically true that you can rank in Google if you block them in robots.txt but it's going to take a lot more work. Also your listing will
Read the rest
look worse (last time I saw this there was no site description, but that was a few years back). If you care about Google SEO traffic you maybe want to let them on your site.
Great article and sleuthing to find the information.
I know you're processing them dynamically as they come in and break the rules. But if you wanted to supplement the list, might be worth sourcing the ones from https://github.com/ai-robots-txt/ai.robots.txt at some frequency.
In Using lots of little tools to aggressively reject the botsI use a robots.txt[^1] to try to prevent them from calling my site. But they're probably doing it regardless ¯\_(ツ)_/¯. Don't want to completely kill my site from indexing, but if I did I'd 100% set up a tarpit.
[^1]: https://github.com/ai-robots-txt/ai.robots.txt
In The benefits of learning in publicI’ve been using https://github.com/ai-robots-txt/ai.robots.txt on my Gitea instance.
In Ask HN: AI bots everywhere – does anyone have a good whitelist for robots.txt?Linked upthread: https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...
In Amazon's AI crawler is making my Git server unstableAuthor of a similar tool here[0]. There are a few implementations of this sort of thing that I know of. Mine is different in that the primary purpose is to slightly alter content statically using a Markov generator, mainly to make it useless for content reposters, secondarily to make it useless to LLM crawlers that ignore my robots.txt file[1]. I assume the generated text is bad enough that the LLM crawlers just throw the result out. Other than the extremely poor quality of the text, my tool doesn't leave any fingerprints (like recursive non-sense links.) In any case, it can be run on static sites with no server-side dependencies so long as you have a way to do content redirection based on User-Agent, IP, etc.
My tool does have a second component - linkmaze - which generates a bunch of nonsense text with a Markov generator, and serve
Read the rest
s infinite links (like Nepthenes does) but I generally only throw incorrigible bots at it (and, at others have noted in-thread, most crawlers already set some kind of limit on how many requests they'll send to a given site, especially a small site.) I do use it for PHP-exploit crawlers as well, though I've seen no evidence those fall into the maze -- I think they mostly just look for some string indicating a successful exploit and move on if whatever they're looking for isn't present.
But, for my use case, I don't really care if someone fingerprints content generated by my tool and avoids it. That's the point: I've set robots.txt to tell these people not to crawl my site.
In addition to Quixotic (my tool) and Napthenes, I know of:
* https://github.com/Fingel/django-llm-poison
* https://codeberg.org/MikeCoats/poison-the-wellms
* https://codeberg.org/timmc/marko/
0 - https://marcusb.org/hacks/quixotic.html
1 - I use the ai.robots.txt user agent list from https://github.com/ai-robots-txt/ai.robots.txt
For the "good" bots which at least respect robots.txt you can use this list to get ahead of them before they pummel your site.
https://github.com/ai-robots-txt/ai.robots.txt
There's no easy solution for bad bots which ignore robots.txt and spoof their UA though.
In Nepenthes is a tarpit to catch AI web crawlersThere's also popular repository that maintains a comprehensive list of LLM and AI related bots to aid in blocking these abusive strip miners.
https://github.com/ai-robots-txt/ai.robots.txt
In AI companies cause most of traffic on forumsIt won't help with the more egregious scrapers, but this list is handy for telling the ones that do respect robots.txt to kindly fuck off:
https://github.com/ai-robots-txt/ai.robots.txt
In AI companies cause most of traffic on forumsBearblog is simple and easy for personal blogging. For something more advanced, you can't go wrong with Ghost (self-hosted or cloud).
To block AI bots, you'll need to add known AI bots[1] to your blog's robots.txt file [2].
[1] https://github.com/ai-robots-txt/ai.robots.txt/blob/main/rob...
[2] https://docs.bearblog.dev/neat-bear-features/#edit-robotstxt or https://ghost.org/help/modifying-robots-txt/
In Ask HN: Where Do Thou BlogShowing the 15 most recent of 22 sources. Enable JavaScript to search and browse every indexed mention.