remote-browser

I'm one of the authors of a browser automation framework called remote-browser that's built on top of the Web Extensions API [1]. The meaning behind the name is that you have access to a Web Extension API browser object in the client environment that gives you remote access to the full API. That offers enough power to do complex browser automation tasks along the same lines of what you can do with Selenium or Puppeteer, but simply using vanilla JavaScript and the Web Extensions API instead of tons of browser-specific implementation code and a custom automation API. If Firefox follows Chrome's lead on removing executeScript, then it will effectively kill the remote-browser project. I find it very unfortunate that the capabilities of browser extensions are being crippled. I understand that there are significant security issues with browser

Ah, thanks, now I get it: it doesn't bother them that fingerprinting lets them be tracked; it bothers them that fingerprinting (and VPN IPs) mean their scrapers get hit with captchas. FWIW, it takes very little effort to completely conceal the fact that firefox is running headless under marionette/webdriver/geckodriver. Chrom(ium) takes more effort, but these guys have solved it (and built a business around it): https://intoli.com/blog/making-chrome-headless-undetectable/ https://github.com/intoli/remote-browser Of course neither of these address fingerprinting -- all your scrape requests will have the same or similar fingerprint, which will lead to captchas pretty quickly. This might help (and might even be part of the reason for buying piles of cellphones): https://news.ycombinator.com/item?id=25379342

Nice! We built something like this 3 years ago while I was still at Intoli. Instead of Puppeteer we used our own Web Extensions API based framework called Remote Browser [1], the core of which was written by my cofounder. The tour is still up at [2]. The servers that actually run the Remote Browser have since gone down, but interestingly you can still run the tour. That's because if you don't change the code in the REPL window, you get cached results (except step 7/7 which scrapes Hacker News and won't work). To get those results, we built a little tour "recorder" that would be run on every release. If I remember correctly, we allowed some dynamic ES6 imports through a custom Babel compiler for the code that's input, which also allows first level async stuff, which still works :) [1]: https://github.com/intoli/remote-browser [2]: https://