BRbryanrasmussen·Jul 29, 2022
first off I would probably want to use price as the starting point, if price is there and the contents of price when trimmed etc. is a decimal, cool.
But if there isn't a .price because someone has changed it to .amount for some reason. Or hey started using Material-UI or some other framework and they no longer have any meaningful classes anywhere which is what I most often see people whining about.
then you would probably get an absolute xpath to the item you want, then you would use the getRobustXPath(element, document) function for that element, using the linked library under discussion https://github.com/cyluxx/robula-plus to get a robust xpath. This xpath will probably be pretty unreadable but you can then test to see it gets your stuff.
How will it be more robust - well that depends but if I have something like this
<div class=ar
Read the rest
ticle-section>
<div>Hammer</div>
<div>
Buy this and everything will start to look like a nail
</div>
</div>
<div class=price>123.45</div>
</div>
I might first off want to get products and their descriptions and their prices together. And I would probably say hey it might be that they are under this kind of class based structure but that can change but what is less likely to change is that a 'price' will be
A number with either . or , as the potential decimal delimiter, with potential currency symbols either directly in the element with the number, not found, or in a directly preceding or following element.
It would be useful to find prices on the page before everything else, and from there to figure out where product names and where descriptions where in relation to the price.
Dependent on language of page product layout will probably change, but in a western language the price will probably come 'last' in our product container, which is exactly what you have done here.
A non-robust xpath for matching the above might look something like
product = //[local-name()='article' || local-name()='main']//[child::*[last()][xs:decimal(text())]
The above might have an error in the xpath because hey, writing without testing in environment, and also obviously really expensive as have not optimized. But it should also be clear enough that by removing the need to have matching elements be based on the incidentals of how we style those elements we can add up with data extraction that is less likely to break with site redesigns.
on edit: obv. article or main check would be optimized beforehand, just some sites use both, some use one, and some neither so would have to do all sorts of things first, etc. etc.