mmdbctl

Another solution is to use an MMDB (“MaxMind DB”) file [1] which is essentially a binary tree + deduplicated values (same as idea 3.1). There are several free ASN MMDBs [2,3] but you can also build your own MMDB files from any Prefix->Value mapping with the mmdbwriter library [4] or a CLI tool built on top of it like mmdbctl [5]. Assuming the ASN MMDB is fully loaded in memory, it would use around 60MB. [1] https://maxmind.github.io/MaxMind-DB/ [2] https://dev.maxmind.com/geoip/docs/databases/asn/ [3] https://ipinfo.io/products/free-ip-data-downloads [4] https://github.com/maxmind/mmdbwriter [5] https://github.com/ipinfo/mmdbctl (I work for IPinfo, but there are lots of other companies offering MMDB files).

https://incolumitas.com/ This is my all-time favorite article: https://incolumitas.com/2021/11/03/so-you-want-to-scrape-lik... I used to do freelance web scraping, and that article felt like some kind of forbidden knowledge. After reading the article, I went down the rabbit hole and actually found a Discord server that provided carrier-grade traffic relay from a van which contained dozens of phones. For the questions..... we have to kinda wait a bit, someone from our engineering team might come here and reply. By the way, as I have you here have you considered converting the CSV files to MMDB format? I was planning to do that with our mmdbctl tool later today. https://github.com/ipinfo/mmdbctl