I'm using a simple urlfetch call in Google App Engine to retrieve pages from a website.
The weird thing is that the pages that are returned are always the "m.website.com" mobile version.
I changed the user agent of the fetch() but nothing seems to change this behavior. Does anyone know whether there is a way to ensure the desktop/laptop web version of the page is returned?
Sample code below:
url = "http://www.example.com/a-page"
urlfetch.set_default_fetch_deadline(25)
page = urlfetch.fetch(url, headers = {'User-Agent': "Mozilla/5.0"})
if page.status_code != 200:
logging.info("alert: URL status code: {}, trying to fetch page: {}".format(page.status_code, url))
if page:
self.response.write("\n{}".format(page.status_code))
self.response.write("\n{}".format(page.final_url))
self.response.write("\n{}".format(page.headers))
self.response.write("\n{}".format(page.header_msg))
soup = BeautifulSoup(page.content)
self.response.write("\n{}".format(soup.prettify(encoding="utf-8")))
the result always takes the first page, "www.example.com/a-page", does a redirect and returns "m.example.com/mobile-version-of-a-page"...