xidel

Hey, wxpath author here. It's pretty cool seeing this project reach the front page a week after posting it. Just wanted to mention a few things. wxpath is a result of a decade of working and thinking about web crawling and scraping. I created two somewhat popular Python web-extraction projects a decade ago (eatiht, and libextract), and even helped publish a metaanalysis on scrapers, all heavily relying on lxml/XPath. After finding some time on my hands and after a hiatus on actually writing web scrapers, I decided to return to this little problem domain. Obviously, LLMs have proven to be quite formidable at web content extraction, but they encounter the now-familiar issues of token limits and cost. Besides LLMs, there's been some great projects making great progress on the problem of web data extraction, like the Scrapy and Crawlee framew

My preferred "self-hosted" webscraper is a local, single binary called xidel [1]. The feature I really like is that it can also follow links. [1] https://github.com/benibela/xidel

Well, it's not as if this is the first free alternative. Here is a wonderful, incredibly powerful tool, not written in Java, but in Free Pascal, which is probably too often underestimated: Xidel[1]. Just have a look at the features and check its Github page[2]. I've often been amazed at its capabilities and, apart from web scraping, I mainly use it for XQuery executions - so far the latest version 0.9.9 has also implemented XPath/XQuery 3.1 perfectly for my requirements. Another insider tip is that XPath/XQuery 3.1 can also be used to transform JSON wonderfully - JSONiq is therefore obsolete. [1] https://www.videlibri.de/xidel.html [2] https://github.com/benibela/xidel

I still like Xidel[0] for this reason; it may be a little older, but for a CLI scraper a lot of data transformations needs can be satisfied with Xpath/XQuery. [0]: https://github.com/benibela/xidel

Also: https://github.com/benibela/xidel

Xidel for XPath on JSON, XML, and HTML: https://github.com/benibela/xidel

> Well, jq is grep as well as sed and awk, but yeah, htmlq seems to be just grep, for sake of comparison. Exactly, and that is what I mean. If you want to compare, compare it with grep, not jq. Someone else posted xidel[0] in this thread, which I've not used, but it seems to be the "jq but for html". [0] https://github.com/benibela/xidel

It's not novel obviously. I have been using pup[1] for years. And xidel[2] is probably older. [1] https://github.com/ericchiang/pup [2] https://github.com/benibela/xidel

This looks very useful, big fan of all the ^[a-z]+q$ utilities out there. But as a user, I would probably want to use XPath[0] notation here. Maybe that is just me. A quick search revealed xidel[1] which seems to be similar, but supports XPath. [0]https://en.wikipedia.org/wiki/XPath [1]https://github.com/benibela/xidel

I generally use xidel [1] for that type of task. Feed it xpath, css selectors or its own pattern matching thing. [1] https://github.com/benibela/xidel

GitHub repo: https://github.com/benibela/xidel