We want to create a java crawler (crawler4j) which uses many product EANs to collect informations like price, picture, description of products from some defined webshops in cooperation with the host of the webshops. These informations should be stored in a MSSQL database. By now we have successfully done some tests about crawling images from websites.
The point is, that the html-pages, which the webshop sends back to the browser, doesnt contain the ean. But the webshop generates the html-code for every request new.
So it is basically possible to get another content based on the requests User-Agent-String?? Should we only have to include "bot" in the user agent string or what is the common way to get the content we want?