How-To-Prevent-Scraping

This is a good article I've followed in the past. idk how helpful this is, you might have already gone through it. https://github.com/JonasCz/How-To-Prevent-Scraping

I assume you are concerned about crawlers that do not respect the robots.txt file (which is the polite way to restrict them from indexing your side, but does not provide any actual protection if crawlers chose to ignore the file). Cloudflare has a tool for doing this (now part of their core service): https://blog.cloudflare.com/introducing-scrapeshield-discove... There's a nice Github repo with some advice on blocking scrapers: https://github.com/JonasCz/How-To-Prevent-Scraping Finally, you could use a plugin in your Webserver to display a CAPTCHA to visitors from IP addresses that cause a lot of requests to your site. There are many more strategies available (up to creating fake websites / content to lead crawlers astray), but the CAPTCHA solution is the most robust one. It will not be able to protect you against crawlers that use a larg