undetected-chromedriver

s downloaded, and so on. My bank isn't giving CAPTCHAs so that would still work if I were to get around to updating my script to deal with some redesigns they did of their pages which broke finding the relevant elements on the page. But I've found that if I do visit a site that uses hCaptcha while using the Selenium launched browser, it seems to get stuck. Click to tell it I'm not a bot. Then get an image test. Answer that correctly and get another image test. Answer that correctly. Then it goes back to the click if you are not a bot thing, and repeats--two more image tests and back to the beginning. Here's a program if anyone wants to try this and has the Selenium Webdriver package for Python3 installed. This will open a browser and take you to fanfiction.net. Trying to actually read any story will bring up the CAPTCHA. #!/usr/bin/env python3 from selenium.webdriver import Chrome driver = Chrome() driver.get("https://www.fanfiction.net") input("press enter when done") driver.close() driver.quit() I'm not sure if the looping is a Cloudflare thing or a fanfiction.net thing, because the latter is the only site I use that has Cloudflare's CAPTCHA. It used to be that if you added from selenium.webdriver import ChromeOptions and changed opening the driver to options = ChromeOptions() options.add_experimental_option("excludeSwitches", ["enable-automation"]) options.add_experimental_option('useAutomationExtension', False) options.add_argument("--disable-blink-features=AutomationControlled") driver = Chrome(options=options) you could get past the CAPTCHA, but that stopped working a while ago. There's this project to provide a Selenium Chrome driver that is supposed to not trigger anti-bot detectors [1], but it still hit the CAPTCHA loop when I tried it. [1] https://github.com/ultrafunkamsterdam/undetected-chromedrive...

=AutomationControlled") A while later though the site started occasionally bringing up that Cloudflare "Checking your browser before accessing <site_name>" automatic check. That just sits there, occasionally reloading. Same thing if I use undetected-chromedriver [1] which is supposed to be less likely to trigger anti-bot services. Next up was using a bookmarklet. It was easy to make a bookmarklet that used an XMLHttpRequest to post the content of document.documentElement.innerHTML to a server of mine for processing. That bookmarklet could then advance to the next page of interest. It wouldn't be as nice as the prior solutions because I'd have to click the bookmarket for every page, but that wouldn't be too bad. The bigger annoyance there is if my server isn't HTTPS the XMLHttpRequest is blocked because of mixed content. In Chrome you can tell it to ignore that on a per site basis, but it isn't persistent across browser launches. There are also issues if the server is on my LAN, which runs into the "not let JavaScript on a page from the internet access the LAN because it might be trying to hack your IoT" stuff. I was just starting to look into browser extensions to see if that would be a viable approach. I don't know if (1) an extension can save to the local filesystem (which I'd prefer instead of having to do that XMLHttpRequest runaround), or (2) if an extension can carry out actions on multiple pages without user intervention (if I still would have to do the click per page I can just stick with the bookmarklet approach). I was even contemplating trying to make a fork of Chromium or Firefox with built-in support for this stuff. Then I remembered AppleScript, and that things that support AppleScript can also be controlled with JavaScript which is much much much less weird than AppleScript. Here's some JXA code that saves current tab's source from Safari: var cur = Application.currentApplication() cur.includeStandardAdditions = true; var app = Application('Safari'); app.includeStandardAdditions = true; function save_page(file_name) { let source = app.windows[0].currentTab.source() let path = Path(file_name) let f = cur.openForAccess(path, {writePermission: true}) cur.setEof(f, {to: 0}) cur.write(source, {to: f, startingAt: 0}) cur.closeAccess(f) } save_page("/tmp/page.html") Here are the changes to do that for Chrome (sort of...it doesn't quite get the full source): 3c3 < var app = Application('Safari'); --- > var app = Application('Google Chrome'); 8c8,9 < let source = app.windows[0].currentTab.source() --- > let ct = app.windows[0].activeTab > let source = app.execute(ct, {javascript: "document.documentElement.innerHTML"}) Changing pages is also easy. For Safari, app.windows[0].currentTab.url = "..." I don't remember what it is for Chrome, but it is similarly easy. And on both there is also the option of injecting JavaScript onto the page to do the navigation, if for some reason just setting the URL isn't good enough. (No Firefox examples because Firefox has absolutely abysmal AppleScript support). With JXA then I can write a handful of short command line utilities that can get the current URL, get the current page content, and navigate to a given URL and I've got everything I need. (There might be a way to sort of make it work with Firefox. I believe (but am not certain) that with JXA I can tell the system send mouse-click events to anywhere I want in an application window. Firefox does provide enough AppleScript support to get the title of the current page. So maybe the bookmarklet approach, with JXA watching for the page changes and clicking the bookmarklet, would work). [1] https://github.com/ultrafunkamsterdam/undetected-chromedrive...