cloudscraper

> Cloudflare can always spot crawlers I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-an... Plenty of open-source ones as well that could bypass, eg maybe this one that came up in search https://github.com/VeNoMouS/cloudscraper Combine with residential proxies and you're just not going to find them. > Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot. Just from a quick Google search: - LinkedIn: https://brightdata.com/products/datasets/linkedin - Amazon: https://www.junglescout.com/features/product-database/

this won't stop the overall trend, but it can help you get around cloudflare's effective scraping blocking (copying my comment from a previous thread): If you're scraping with Python, try cloudscraper—among other things(!), it supports JS rendering (basically the bare-minimum check cloudflare does), without needing to run a full browser in the background. It's built on requests, so integration was pretty easy. https://github.com/venomous/cloudscraper

If you're scraping with Python, try cloudscraper—among other things(!), it supports JS rendering (basically the bare-minimum check cloudflare does), without needing to run a full browser in the background. It's built on requests, so integration (for me, anyway) was pretty easy. https://github.com/venomous/cloudscraper

Anyone with a stake in bypassing anti-bot measures isn't going to share their tactics, since sharing it will lead to such workaround being patched or mitigated, requiring them to research for more bot detection workarounds. Projects like cloudscraper[0] are often linked to point and say "look! they broke Cloudflare!" but CF and the rest of the industry has detections for tools like this, and instead of rolling out blocks for these tools, they give website owners tools like bot score[1] to manage their own risk level on a per-page basis. 0: https://github.com/VeNoMouS/cloudscraper 1: https://developers.cloudflare.com/bots/concepts/bot-score/

I'm really surprised that the JS challenges helped so much, given that there are open source libraries for bypassing them (e.g. cloudscraper[0]). [0]: https://github.com/venomous/cloudscraper

I think javascript comes up because cloudflare use some kind of javascript challenge as part of the DDOS protection. There are python libraries that know how to deal with it, or you can use some level of headless browser. https://github.com/VeNoMouS/cloudscraper