mlscraper

Why GPT-based then? There are libraries that do this: You give examples, they generate the rules for you and give you a scraper object that takes any html and returns the scraped data. Mine: https://github.com/lorey/mlscraper Another: https://github.com/alirezamika/autoscraper

Personally, this feels like the direction scraping should move into. From defining how to extract, to defining what to extract. But we're nowhere near that (yet). A few other thoughts from someone who did his best to implement something similar: 1) I'm afraid this is not even close to cost-effective yet. One CSS rule vs. a whole LLM. A first step could be moving the LLM to the client side, reducing costs and latency. 2) As with every other LLM-based approach so far, this will just hallucinate results if it's not able to scrape the desired information. 3) I feel that providing the model with a few examples could be highly beneficial, e.g. /person1.html -> name: Peter, /person2.html -> name: Janet. When doing this, I tried my best at defining meaningful interfaces. 4) Scraping has more edge-cases than one can imagine. One example being nest

Really interesting tool. I've always found the task of finding the right CSS selectors tiresome, so I created tool which automatically extracts the right selectors for you (and builds a scraper as well). Open Source on GitHub: https://github.com/lorey/mlscraper

In case anyone wants to detect the selectors automatically, here's a small python library I wrote that does it for you: https://github.com/lorey/mlscraper