gae urlfetch for some urls returns the mobile version of the url

139 views Asked by At

I'm using a simple urlfetch call in Google App Engine to retrieve pages from a website.

The weird thing is that the pages that are returned are always the "m.website.com" mobile version.

I changed the user agent of the fetch() but nothing seems to change this behavior. Does anyone know whether there is a way to ensure the desktop/laptop web version of the page is returned?

Sample code below:

    url = "http://www.example.com/a-page"
    urlfetch.set_default_fetch_deadline(25)
    page = urlfetch.fetch(url, headers = {'User-Agent': "Mozilla/5.0"})
    if page.status_code != 200:
        logging.info("alert: URL status code: {}, trying to fetch page: {}".format(page.status_code, url))
    if page:
        self.response.write("\n{}".format(page.status_code))
        self.response.write("\n{}".format(page.final_url))
        self.response.write("\n{}".format(page.headers))
        self.response.write("\n{}".format(page.header_msg))
        soup = BeautifulSoup(page.content)
        self.response.write("\n{}".format(soup.prettify(encoding="utf-8")))

the result always takes the first page, "www.example.com/a-page", does a redirect and returns "m.example.com/mobile-version-of-a-page"...

0

There are 0 answers