HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
ai-robots-txt
ai.robots.txt
Python2024–2026
ai-robots-txt / ai.robots.txt

ai.robots.txt

A list of AI agents and robots to block.

17Distinct people
22Mentions
0Explicit recommenders
4.3KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A list of AI agents and robots to block.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 7:57:46 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month22 dated mentions

Observed mentions: 2024-03-28 11:46:09 UTC to 2026-02-11 22:48:21 UTC.

0123452024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsMar 20242024-04-01 00:00:00 UTC to 2024-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-05-01 00:00:00 UTC to 2024-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-06-01 00:00:00 UTC to 2024-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-07-01 00:00:00 UTC to 2024-08-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2024-08-01 00:00:00 UTC to 2024-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments2024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 5 mentions; 0 submissions, 5 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsFeb 20252025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-12-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2026-01-01 00:00:00 UTC to 2026-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-02-01 00:00:00 UTC to 2026-03-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsFeb 2026

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC110
Apr 20242024-04-01 00:00:00 UTC2024-05-01 00:00:00 UTC000
May 20242024-05-01 00:00:00 UTC2024-06-01 00:00:00 UTC000
Jun 20242024-06-01 00:00:00 UTC2024-07-01 00:00:00 UTC000
Jul 20242024-07-01 00:00:00 UTC2024-08-01 00:00:00 UTC202
Aug 20242024-08-01 00:00:00 UTC2024-09-01 00:00:00 UTC000
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC312
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC000
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC101
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC303
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC505
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC101
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC101
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC000
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC101
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC000
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC101
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC110
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC000
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC000
Dec 20252025-12-01 00:00:00 UTC2026-01-01 00:00:00 UTC101
Jan 20262026-01-01 00:00:00 UTC2026-02-01 00:00:00 UTC000
Feb 20262026-02-01 00:00:00 UTC2026-03-01 00:00:00 UTC101

What Hacker News says

5 sources
Author of a similar tool here[0]. There are a few implementations of this sort of thing that I know of. Mine is different in that the primary purpose is to slightly alter content statically using a Markov generator, mainly to make it useless for content reposters, secondarily to make it useless to LLM crawlers that ignore my robots.txt file[1]. I assume the generated text is bad enough that the LLM crawlers just throw the result out. Other than the extremely poor quality of the text, my tool doesn't leave any fingerprints (like recursive non-sense links.) In any case, it can be run on static sites with no server-side dependencies so long as you have a way to do content redirection based on User-Agent, IP, etc. My tool does have a second component - linkmaze - which generates a bunch of nonsense text with a Markov generator, and serve
Read the rest
s infinite links (like Nepthenes does) but I generally only throw incorrigible bots at it (and, at others have noted in-thread, most crawlers already set some kind of limit on how many requests they'll send to a given site, especially a small site.) I do use it for PHP-exploit crawlers as well, though I've seen no evidence those fall into the maze -- I think they mostly just look for some string indicating a successful exploit and move on if whatever they're looking for isn't present. But, for my use case, I don't really care if someone fingerprints content generated by my tool and avoids it. That's the point: I've set robots.txt to tell these people not to crawl my site. In addition to Quixotic (my tool) and Napthenes, I know of: * https://github.com/Fingel/django-llm-poison * https://codeberg.org/MikeCoats/poison-the-wellms * https://codeberg.org/timmc/marko/ 0 - https://marcusb.org/hacks/quixotic.html 1 - I use the ai.robots.txt user agent list from https://github.com/ai-robots-txt/ai.robots.txt
In Nepenthes is a tarpit to catch AI web crawlers

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.