crawlee

Problem with fixed concurrency is that required memory per Chrome process varies a lot. We (https://www.apify.com) are solving this by autoscaling number of parallel Puppeteer instances based on memory and CPU. Our open source SDK (http://github.com/apifytech/apify-js) implements this using class PuppeteerCrawler (https://www.apify.com/docs/sdk/apify-runtime-js/latest#Puppe...) which internally uses AutoscaledPool that provides autoscaling: - https://github.com/apifytech/apify-js/blob/master/src/autosc... - https://www.apify.com/docs/sdk/apify-runtime-js/latest#Autos... Sadly this feature is currently limited to our platform because it's mainly build for running in Docker containers. And as Docker container don't know about it's CPU consumption vs limits it requires a notifications about reaching its CPU limit from underlying platform. We

> we have developed a solution which removes the property from the web browser and thus prevents these kind of protections from figuring out that the browser is automated Eli Grey and I have bypassed your "hideWebDriver()" function[1] in a single line of code: if (navigator.webdriver || (Navigator.prototype && Object.getOwnPropertyDescriptors(Navigator.prototype)["webdriver"])) { // Chrome headless detected - navigator.webdriver exists or was redefined } [1]: https://github.com/apifytech/apify-js/blob/262a2e604b1adb3d8...