ai-bot-blocker

First off, I want to thank you and the other members of the CC Foundation, the CC data set is an incredible resource to everyone. Much of the UA data, including CCBot, is from an upstream source[0]. I was torn on whether CCBot and other archival bots should be included in the configs, since these services are not AI bot scraping services. I've added an exclusion for CCBot[1] and the archival services from the recommended configs. [0] https://darkvisitors.com/agents/ccbot [1] https://github.com/anthmn/ai-bot-blocker/commit/ae0c2c40fd08...

For those not using cloudflare but who have access to web server config files and want to block AI bots, I put together a set of prebuilt configs[0] (for Apache, Nginx, Lighttpd, and Caddy) that will block most AI bots from scraping contents. The configs are built on top of public data sources[1] with various adjustments. [0] https://github.com/anthmn/ai-bot-blocker [1] https://darkvisitors.com/

Hello HN. I recently released the source code for a set of AI / LLM bot blocker web server configs that are used by my search engine (Ichido Search [0]) to block AI bots from crawling the pages. Sharing here, as others may have use cases for these configs as well. Supports the following web servers: * Apache (and .htaccess) * Nginx * Lighttpd * Caddy * IIS (untested) [0] https://ichi.do/