HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
TecharoHQ
anubis
Go2025–2026
TecharoHQ / anubis

anubis

Weighs the soul of incoming HTTP requests to stop AI crawlers

67Distinct people
80Mentions
1Explicit recommenders
24.1KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Weighs the soul of incoming HTTP requests to stop AI crawlers

GitHub

Where it is used

The last source check was inconclusive. This does not establish whether a product uses it.

What people are saying

This is just like security; the most secure system is the one that nobody can use. I think the proof-of-work approach that anubis[0] takes is pretty interesting. I love the…

wolttam used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 8:59:02 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month80 dated mentions

Observed mentions: 2025-03-18 05:45:12 UTC to 2026-10-18 18:22:03 UTC.

051015202025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 12 mentions; 3 submissions, 9 commentsMar 20252025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 6 mentions; 0 submissions, 6 comments2025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 5 mentions; 0 submissions, 5 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 4 mentions; 1 submissions, 3 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 20 mentions; 1 submissions, 19 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 5 mentions; 0 submissions, 5 comments2025-12-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 commentsDec 20252026-01-01 00:00:00 UTC to 2026-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-02-01 00:00:00 UTC to 2026-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-03-01 00:00:00 UTC to 2026-04-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2026-04-01 00:00:00 UTC to 2026-05-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2026-05-01 00:00:00 UTC to 2026-06-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2026-06-01 00:00:00 UTC to 2026-07-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments2026-07-01 00:00:00 UTC to 2026-08-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2026-08-01 00:00:00 UTC to 2026-09-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments2026-09-01 00:00:00 UTC to 2026-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2026-10-01 00:00:00 UTC to 2026-11-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 commentsOct 2026

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC1239
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC606
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC505
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC211
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC413
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC20119
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC101
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC101
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC505
Dec 20252025-12-01 00:00:00 UTC2026-01-01 00:00:00 UTC404
Jan 20262026-01-01 00:00:00 UTC2026-02-01 00:00:00 UTC000
Feb 20262026-02-01 00:00:00 UTC2026-03-01 00:00:00 UTC000
Mar 20262026-03-01 00:00:00 UTC2026-04-01 00:00:00 UTC101
Apr 20262026-04-01 00:00:00 UTC2026-05-01 00:00:00 UTC211
May 20262026-05-01 00:00:00 UTC2026-06-01 00:00:00 UTC202
Jun 20262026-06-01 00:00:00 UTC2026-07-01 00:00:00 UTC404
Jul 20262026-07-01 00:00:00 UTC2026-08-01 00:00:00 UTC303
Aug 20262026-08-01 00:00:00 UTC2026-09-01 00:00:00 UTC404
Sep 20262026-09-01 00:00:00 UTC2026-10-01 00:00:00 UTC101
Oct 20262026-10-01 00:00:00 UTC2026-11-01 00:00:00 UTC303

What Hacker News says

20 sources
Those servers are mainly designed for enterprise use cases. For hobby projects, I can understand why someone would choose Hetzner over AWS. For enterprise environments, however, there is much more to consider. One of the biggest costs you face is your operations team. If you go with Hetzner, you essentially have to rebuild a wide range of infrastructure components yourself (WAF, globally distributed CDN, EFS, RDS, EKS, Transit Gateways, Direct Connect and more). Of course, you can create your own solutions for all of these. At my company, a mid-size enterprise, we once tried to do exactly that. WAF: https://github.com/TecharoHQ/anubis CDN: Hetzner Nodes with Cache in Finnland, USA and GER RDS: Self-hosted MySQL from Bitnami EFS: https://github.com/rook/rook EKS: https://github.com/vitobotta/hetzner-k3s and 20+ more moving targets of infra
Read the rest
software stack and support systems The result was hiring more than 10 freelancers in addition to 5 of our DevOps engineers to build it all and handling the complexity of such a setup and the keep everything up-to-date, spending hundreds of thousands of dollars. Meanwhile, our AWS team, consisting of only three people working with Terraform, proved far more cost-effective. Not in terms of dollars per CPU core, but in terms of average per project spending dollars once staff costs and everything were included. I think many of the HN posts that say things like "I saved 90% of my infra bill by moving from AWS to a single Hetzner server" are a bit misleading.
In Use One Big Server (2022)
CDNs like Cloudflare are the best. Anubis is a rate limitor for small websites where you can't or won't use CDNs like Cloudflare. I have used Cloudflare on several medium sized websites and it works really well. Anubis's creator says the same thing: > In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you. Source: https://github.com/TecharoHQ/anubis
In AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders
For what it's worth, kernel.org seems to be running an old version of Anubis that predates the current challenge generation method. Previously it took information about the user request, hashed it, and then relied on that being idempotent to avoid having to store state. This didn't scale and was prone to issues like in the OP. The modern version of Anubis as of PR https://github.com/TecharoHQ/anubis/pull/749 uses a different flow. Minting a challenge generates state including 64 bytes of random data. This random data is sent to the client and used on the server side in order to validate challenge solutions. The core problem here is that kernel.org isn't upgrading their version of Anubis as it's released. I suspect this means they're also vulnerable to GHSA-jhjj-2g64-px7c.
In Why are anime catgirls blocking my access to the Linux kernel?
Wikipedia says their traffic increased roughly 50% [1] from AI bots, which is a lot, sure, but nowhere near the amount where you'd have to rearchitect your site or something. And this checks out, if it was actually debilitating, you'd notice Wikipedia's performance degrade. It hasn't. You'd see them taking some additional steps to combat this. They haven't. Their CDN handles it just fine. They don't even both telling AI bots to just download the tarballs they specifically make available for this exact use case. More importantly, Wikipedia almost certainly represents the ceiling of traffic increase. But luckily, we don't have to work with such coarse estimation, because according to Cloudflare, the total increase from combined search and AI bots in the last year (May 2024 - May 2025), has just been... 18% [2]. The way you hear people talk
Read the rest
about it though, you'd think that servers are now receiving DDOS-levels of traffic or something. For the life of me I have not been able to find a single verifiable case of this. Which if you think about it makes sense... It's hard to generate that sort of traffic, that's one of the reasons people pay for botnets. You don't bring a site to its knees merely by accidentally "not making your scraper efficient". So the only other possible explanation would be such a larger number of scrapers simultaneously but independently hitting sites. But this also doesn't check out. There aren't thousands of different AI scrapers out there that in aggregate are resulting in huge traffic spikes [2]. Again, the total combined increase is 18%. The more you look into this accepted idea that we are in some sort of AI scraping traffic apocalypse, the less anything makes sense. You then look at this Anubis "AI scraping mitigator" and... I dunno. The author contends that one if its tricks is that it not only uses JavaScript, but "modern JavaScript like ES6 modules," and that this is one of the ways it detects/prevents AI scrapers [3]. No one is rolling their own JS engine for a scraper such that they are being blocked from their inability to keep up with the latest ECMAScript spec. You are just using an existing JS engine, all of which support all these features. It would actually be a challenge to find an old JS engine these days. The entire things seems to be built on the misconception that the "common" way to build a scraper is doing something curl-esque. This idea is entirely based on the google scraper which itself doesn't even work that way anymore, and only ever did because it was written in the 90s. Everyone that rolls their own scraper these days just uses Puppeteer. It is completely unrealistic to make a scraper that doesn't run JavaScript and wait for the page to "settle down" because so many pages, even blogs, are just entirely client-side rendered SPAs. If I were to write a quick and dirty scraper today I would trivially make it through Anubis' protections... by doing literally nothing and without even realizing Anubis exists. Just using standard scraping practices with Puppeteer. Meanwhile Anubis is absolutely blocking plenty of real humans, with the author for example telling people to turn on cookies so that Anubis can do its job [4]. I don't think Anubis is blocking anything other than humans and Message's link preview generator. I'm investigating further, but I think this entire thing may have started due to some confusion, but want to see if I can actually confirm this before speculating further. 1. https://www.techspot.com/news/107407-wikipedia-servers-strug... (notice the clickbait title vs. the actual contents) 2. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-cr... 3. https://codeberg.org/forgejo/discussions/issues/319#issuecom... 4. https://github.com/TecharoHQ/anubis/issues/964#issuecomment-...
In Robots.txt is a suicide note (2011)
Try opening a bug report? https://github.com/TecharoHQ/anubis?tab=readme-ov-file#suppo... Compared to Cloudflare and Google, you can actually talk to a human here and they might (I have no reason to believe the opposite) actually care about niche browsers whereas Google seems to test their products on browsers other than their own engine only after release (presumably people will get annoyed about these breakages/outages eventually and switch to a their browser or a rebrand thereof). There's no such conflict of interest here. I'm not aware of a better thing to use than Anubis or a self-written equivalent (my understanding is that it's a simple sha2 PoW)
In FFmpeg moves to Forgejo

Showing the 15 most recent of 20 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.