blocklist-ipsets

# verify you have console access to the server before doing things like this: for BadIP in $(grep -Ev ^# ipip_country_br.netset); do ip route add blackhole "${BadIP}" 2>/dev/null;done Example of country IP addresses for Brazil [2] As this is academia I don't know if there is a concept of service level agreement or a promise of availability, but during attacks the requests to specific URL's could be redirected to a static pre-compressed landing page served out of memory that says "Access to these documents limited during AI bot attacks, here is where to request a full download instead: " I forgot to mention, many of the AI bots are limited to HTTP/1.1. During an attack your server could redirect those request to a static page as well. Curious if you can tell by your access logs if the majority of the attack is HTTP/1.1 or HTTP/2.0. Most botters are too lazy to implement HTTP/2.0. Real browser clients use HTTP/2.0. Example of using HTTP/1.1 and HTTP/2.0 # Fist use HTTP/2.0 with headers I expect. curl -i --http2 \ -A "Mozilla/5.0 (X11; Linux x86_64; rv:151.0) Gecko/20100101 Firefox/151.0" \ -H "sec-fetch-mode: navigate" \ -H "accept-language: en-US,en;q=0.9" \ --url 'https://blawg.nochan.net/.env' # Then HTTP/1.1 for the lulz, pay attention to the Location redirect curl -i --http1.1 \ -A "Mozilla/5.0 (X11; Linux x86_64; rv:151.0) Gecko/20100101 Firefox/151.0" \ -H "sec-fetch-mode: navigate" \ -H "accept-language: en-US,en;q=0.9" \ --url 'https://blawg.nochan.net/.env' Some bots also do not hide that they are bots in that they say they are bot in their user-agent client header. That would be too easy so I doubt that is the case. [1] - https://github.com/firehol/blocklist-ipsets/ [2] - https://github.com/firehol/blocklist-ipsets/blob/master/ipip...

ld have included this node as well. Here [3] are a few shell functions for getting some of the cloud CIDR blocks. I must have been inebriated when I wrote those. This site may not be reachable during blood moons or when the nanosecond is divisible by zero. Here [4a][4b] are a couple decent subnet calculators. There are some command line tools for playing with CIDR blocks and IP addresses to see if an IP is included in a CIDR block but this varies by Linux distribution so perhaps look for a generic python script. To get a list of Tor exit nodes to blackhole route, look at [5]. This updates often. Just clone the entire repo. Unless your site is related to government dissent or anonymous porn then most traffic from Tor exit nodes will likely just be bots and thus riff-raff. Seconds after I linked realhackers bots showed up and got a zero byte response. Poor lil HN servers must get a lot of trash non stop. I hope I get some delicious bots today. [1] - https://bgp.tools/ [2] - https://bgp.tools/as/14618 [3] - https://ai.realhackers.org/_get_cloud_cidr.txt [4a] - https://mxtoolbox.com/subnetcalculator.aspx [4b] - https://www.vultr.com/resources/subnet-calculator/ [5] - https://github.com/firehol/blocklist-ipsets/blob/master/clea...

to also have a URL your visitors can go to in order to grab their IP address should they claim they are blocked. Link to this comment in a Jira and test it in a staging environment. Be sure to test all the regular outbound connections that you make in production in your staging environment. If you do want to block entire countries without having to look something up on every web hit then use this [4] to add to the blackhole script with a little data massaging. Everything in the blackhole text file must be versioned and committed into your internal repository so one can tell when a network added is breaking things and one can quickly roll back. Have a function in the script that deletes all the blackholes. I blackhole about 27,000 CIDR blocks on a tiny VM web server and the script takes 20 seconds to complete. [1] - https://bgp.tools/ [2] - https://bgp.he.net/ [3] - https://adrianpopagh.blogspot.com/2008/03/route-summarizatio... [4] - https://github.com/firehol/blocklist-ipsets/tree/master/ipip...

a few hours. Be polite and just give them the facts so they don't get defensive. If they don't answer try email. [1] - https://github.com/firehol/blocklist-ipsets.git [2] - https://bgp.he.net/AS4983#_whois

his will block Google bots as they still do not support HTTP/2.0 but will also block a majority of the problematic bots. [1] - https://github.com/firehol/blocklist-ipsets/blob/master/fire... [2] - https://github.com/firehol/blocklist-ipsets

al-host/IP/domains to either write into a tmpfs ram disk or to /dev/null once you get bored watching them. If writing to tmpfs be sure to set up a more frequent log rotation and less retention. Be sure to also set the HTTP keepalive time very low so they are not wasting your resources. No need to tarpit them, most of the code used by these bots will time out after a few seconds. [Edit] I should add, these are not hackers you are messing with. They are just parasitical script-kiddies that prey on the weak, most often using their batches of WordPress hosts as jump-off points and watering hole phishing sites to scam the elderly out of their life savings. Making them lose money would be great too! That's much harder. Most of the IP's you see do not belong to the attacker. The attacker is likely using a set of Command and Control C&C nodes that control a swath of end-user PC's and Wordpress servers infected with malware. Anything you do will cost them nothing but it might slow their threads down a little. The only way to make them lose money would be to find who is controlling the C&C nodes, gather evidence and have law enforcement take them down. To do this one would have to break the law by hacking into the compromised host and see what C&C nodes it is dialing home to, then hack into those hosts and see who is connecting to them. They could be using Tor but most people are lazy enough to not do this. [1] - https://github.com/firehol/blocklist-ipsets.git

e will not take responsibility and will instead get upset with the VPS provider and move to a different provider and this has financial ramifications. Most VPS providers don't have the technical resources to track down the command-and-control nodes. This requires a higher level of visibility they do not have. The folks that have this level of visibility will not likely assist with bot noise unless the bots are also being used to steal government secrets or embarrass government officials or attack public infrastructure. There are ways to trick some of the bots into attacking public infrastructure but I would not want to be caught up in that legal quagmire as I am easy to find and bot owners are not. In summary, the best one can do is either configure systems to discard the noise in web logs and block low reputation IP addresses in their mail servers or if one feels the desire to do something, create tarpits/honeypots for the bots to get stuck in. Tarpits have very little impact on bots but if one wanted to feel like they were doing something it's a start. For example, I have numerous nodes that accept email for all the popular domains. Spam bots think they are relaying thousands of emails through my nodes per day but that is a tiny fraction of a tiny fraction of the spam that people will receive. I am not even making a tiny dent in the problem. As a side note, there are ways to block some of these bots if you control your web and email servers but that is another topic all together and if too many people implemented such techniques then the bot developers would evolve around it. Apologies if this sounds defeatist. Some use blocklists [1][2] to reduce the noise on their web servers and RBL/RSL [3][4][5] servers to reduce the noise on their mail servers but this has limited efficacy and I think it just takes up memory by increasing the routing table or takes up CPU by making large ipset lists for iptables. [1] - https://github.com/firehol/blocklist-ipsets [2] - https://github.com/StevenBlack/hosts [3] - https://www.uceprotect.net/en/index.php?m=6&s=0 [4] - https://www.spamcop.net/fom-serve/cache/291.html [5] - https://www.spamhaus.org/zen/