PHphilipDS·Sep 28, 2012
An alternative to Python or C# for web crawling could be Node.js. It's pretty good at it and you have a few libraries that can help you:
* Node.io (https://github.com/chriso/node.io)
* Phantomjs-node for dynamic content (https://github.com/sgentle/phantomjs-node)
* Cheerio for a jQuery server-side implementation (https://github.com/MatthewMueller/cheerio)
* Node-jquery as an alternative to Cheerio (https://github.com/coolaj86/node-jquery)
A single product might have a different name, but you might try to scrape the product ID's if they exist. Product IDs should be unique. If both websites provide those ID's you could compare those in your database.
If that's not possible: as a small hack you could also use Amazon Mechanical Turk to issue manual tasks to compare product names. This way, real people will check if two products are the same i
Read the rest
n case there is doubt. This will cost you a little, but you could give those people 5 cents per product comparison or something like that.
For question 3, some websites don't allow you to crawl their content. Read their ToS :-)
For the rest I agree with Piskvorrr, you could do some trial and error and learn on the fly or read some books (and still do trial and error and learn on the fly ;-)). Good luck!