burla

i would probably use Playwright with custom code, create chunks based on similar products, then run it on a large cluster in parallel (https://github.com/Burla-Cloud/burla). if you have a single worker trying to scrape a shit ton of products back to back to back you're going to get rate limited or their bot detection will catch you.

Hi all, we're Jake, Jack and Joe, and we're building the open-source alternative to modal labs. We think Modal's great, and wanted to use it at our previous places of work, but weren't allowed to because it was too expensive, and our data couldn't leave our cloud. This is why we built Burla, the open-source alternative to modal. Burla can run any python function on 1000 computers in around 1 second. It works with any container, any hardware, and can be installed with one command (currently only compatible with Google Cloud). Would love to hear any thoughts or suggestions! We're open to chatting: https://cal.com/jakez/burla

Hi HN! We're Jake, Jack, and Joe, and we're happy to share what we think is probably the simplest way to scale python code across thousands of computers in the cloud. Coming from the Biotech world we were frustrated by how surprisingly slow and difficult it can be to iterate on large-scale batch processing pipelines, while contending with the security requirements imposed on us by the FDA. Simple small changes necessitated rebuilding docker containers, waiting for batch pipelines to redeploy, and waiting for vm's to cold boot, a >5minute per iteration dev cycle, all just so I can see what error my code throws this time, then do it all over again! Many other tools in the space were either too complicated for our bioinformaticians and data scientists to use, closed-source / managed only (a no-no with medical data), too difficult to setup an