curl_cffi

+1 to the AMC Metreon being one of the toughest. Of all the theaters I'm monitoring, it's probably in the top 3 of fewest seats opening up. My best guess is that SF is just a market where demand far exceeds supply. But I am seeing a few every day, and I'm guessing it will get better after opening weekend too. My system checks every 10 mins leading up to 24 hours before showing, then ramps up to every 60 seconds. Re: Cloudflare - not sure if you're trying to hit AMC's site directly or an alternative. Either way, the first library I reach for in that situation is curl_cffi https://github.com/lexiforest/curl_cffi

What I have seen it is hard to tell what "serious scrapers" use. They use many things. Some use this, some not. This is what I have learned reading webscraping on reddit. Nobody speaks things like that out loud. There are many tools, see links below Personally I think that running selenium can be a bottle neck, as it does not play nice, sometimes processes break, even system sometimes requires restart because of things blocked, can be memory hog, etc. etc. That is my experience. To be able to scale I think you have to have your own implementation. Serious scrapers complain about people using selenium, or derivatives as noobs, who will come back asking why page X does not work in scraping mechanisms. https://github.com/lexiforest/curl_cffi https://github.com/encode/httpx https://github.com/scrapy/scrapy https://github.com/apify/crawlee

There's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-impersonate There's also Python bindings for the fork for anyone who uses Python: https://github.com/lexiforest/curl_cffi

Any reason you didn’t use https://github.com/lexiforest/curl_cffi?

The same author also makes a Python binding of this which exposes a requests-like API in Python, very helpful for making HTTP reqs without the overhead of running an entire browser stack: https://github.com/lexiforest/curl_cffi I can't help but feel like these are the dying breaths of the open Internet though. All the megacorps (Google, Microsoft, Apple, CloudFlare, et al) are doing their damndest to make sure everyone is only using software approved by them, and to ensure that they can identify you. From multiple angles too (security, bots, DDoS, etc.), and it's not just limited to browsers either. End goal seems to be: prove your identity to the megacorps so they can track everything you do and also ensure you are only doing things they approve of. I think the security arguments are just convenient rationalizations in service of this go