curl-impersonate

There are projects that do this already e.g. https://github.com/lwthiker/curl-impersonate

There are a bunch of these tools out there. I like curl-impersonate[0] as a quick way to make curl fingerprint as chrome or firefox. There's also CylceTLS[1] for a more programmatic approach. [0]: https://github.com/lwthiker/curl-impersonate [1]: https://github.com/Danny-Dasilva/CycleTLS

JA3/JA4 are useless now. At best they identify the family of browser, and spoofing it is table stakes for bad actors. https://github.com/lwthiker/curl-impersonate

Is there a plan to keep this updated? There are a number of projects like curl-impersonate[1] and the like, but these are rarely updated any more. OTOH, I'm curious if people are actually getting by running old versions for production use cases. [1] https://github.com/lwthiker/curl-impersonate

Do you actually use this? $ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html MD5 (How I Block All 26 Million Of Your Curl Requests.html) = e114898baa410d15f0ff7f9f85cbcd9d (downloaded with Safari) $ curl https://foxmoss.com/blog/packet-filtering/ | md5sum e114898baa410d15f0ff7f9f85cbcd9d - I'm aware of curl-impersonate https://github.com/lwthiker/curl-impersonate which works around these kinds of things (and makes working with cloudflare much nicer), but serious scrapers use chrome+usb keyboard/mouse gadget that you can ssh into so there's literally no evidence of mechanical means. Also: If you serve some Anubis code without actually running the anubis script in the page, you'll get some answers back so there's at least one anubis-simulator running on the Internet that doesn't bother to actually run the Jav

Try this then: https://github.com/lwthiker/curl-impersonate

There are tools like curl-impersonate: https://github.com/lwthiker/curl-impersonate out there that allow you to pretend to be any browser you like. Might take a bit of trial and error, but this mechanism could be bypassed with some persistence in identifying what is it that the resource is trying to block.

Curl-impersonate does this https://github.com/lwthiker/curl-impersonate

> Does cURL have an easy-to-spot fingerprint outside of its headers? If it's a https URL: Yes, the TLS handshake. There are curl builds[1] which try (and succeed) to imitate the TLS handshake (and settings for HTTP/2) of a normal browser, though. [1] https://github.com/lwthiker/curl-impersonate

Given that curl-impersonate[1] exists and that a major player in this space is also looking for experience with this library, I'm pretty sure forcing the execution of JS using DOM stuff would be a much more effective deterrent to prevent scraping. [1] https://github.com/lwthiker/curl-impersonate

It says "Firefox(In progress)", and the original project this was forked from has it: https://github.com/lwthiker/curl-impersonate

For mine, I just used a $1/mo VPS with a cron job running a bash script that used curl to poll the parking system and determine if its response was interesting. I used "copy as cURL" from browser dev tools to find something usable (which incorporated a slightly fragile, but very long lived, session token corresponding to my "cart") and I actually had to use curl-impersonate [0] for Cloudflare bot detection to stay out of my way. And then curl to alert me via Pushover. [0] https://github.com/lwthiker/curl-impersonate

Hi HN! After being frustrated to need to run headless chrome/selenium in my small scale scrapers because of Cloudflare and Akamai performing TLS fingerprinting. I decided to create a python library to use chromium's network library called cronet. If you don't need to run javascript and are only affected by TLS fingerprinting, using this library should be more efficient than pupeteer/headeless chrome. The library was inspired by curl-impersonate(https://github.com/lwthiker/curl-impersonate) which solves something similar but by patching curl instead. I think the benefit of python-cronet is that it's going to be simpler to keep it up to date.