How do I stop the 302 url redirection in a simple web crawl?

Question

I am trying to crawl a website using the Requests library in Python, and when I try:

r = requests.get('http://www.cell.com/cell-stem-cell/home', allow_redirects = False)
>>> r.status_code
302
>>> r.text
'The URL has moved <a href="https://secure.jbs.elsevierhealth.com/action/getSharedSiteSession?redirect=http%3A%2F%2Fwww.cell.com%2Fcell-stem-cell%2Fhome&rc=0&code=cell-site">here</a>\n'

and when I try:

>>> r = requests.get("https://secure.jbs.elsevierhealth.com/action/getSharedSiteSession?redirect=http%3A%2F%2Fwww.cell.com%2Fcell-stem-cell%2Fhome&rc=0&code=cell-site")
>>>
>>> r.text
'\n\n\n\n\n<style type="text/css">\n    .hidden {\n        display: none;\n        visibility: hidden;\n    }\n</style>\n\n<!-- hidden iFrame for each of the SSO URLs -->\n<div class="hidden">\n    \n        <iframe src="//acw.secure.jbs.elsevierhealth.com/SSOCore/update?utt=81c120bb854495181ef4ef3f679b12261e956c5-JKh">Your browser doesn\'t support iFrames!</iframe>\n    \n        <iframe src="//acw.sciencedirect.com/SSOCore/update?utt=81c120bb854495181ef4ef3f679b12261e956c5-JKh">Your browser doesn\'t support iFrames!</iframe>\n    \n        <iframe src="//acw.scopus.com/SSOCore/update?utt=81c120bb854495181ef4ef3f679b12261e956c5-JKh">Your browser doesn\'t support iFrames!</iframe>\n    \n        <iframe src="//acw.sciverse.com/SSOCore/update?utt=81c120bb854495181ef4ef3f679b12261e956c5-JKh">Your browser doesn\'t support iFrames!</iframe>\n    \n        <iframe src="//acw.mendeley.com/SSOCore/update?utt=81c120bb854495181ef4ef3f679b12261e956c5-JKh">Your browser doesn\'t support iFrames!</iframe>\n    \n        <iframe src="//acw.elsevier.com/SSOCore/update?utt=81c120bb854495181ef4ef3f679b12261e956c5-JKh">Your browser doesn\'t support iFrames!</iframe>\n    \n</div>\n\n\n\n<noscript>\n    <a href="CANT POST LINK BECAUSE OF LACK OF REPUTATION POINTS OF STACK OVERFLOW">Redirect</a>\n</noscript>\n\n<!-- redirect to the product page after all iFrames are rendered -->\n<script>\n    setTimeout(redirectFun,2000);\n    var iFramesList = document.getElementsByTagName("iframe");\n    var renderedIFramesCount = 0;\n    var numberOfIFrames = iFramesList.length;\n    for (var i = 0; i < iFramesList.length; i++) {\n        var iFrame = iFramesList[i];\n        bindEvent(iFrame, \'load\', function(){\n            renderedIFramesCount = renderedIFramesCount + 1;\n            if (renderedIFramesCount >= numberOfIFrames)\n            {\n                redirectFun();\n            }\n        });\n    }\n    var doRedirect = true;\n    function redirectFun() {\n        if (doRedirect)\n            window.location.href = "CANT POST THIS WEBSITE BECAUSE OF MY REPUTATION POINTS ON STACKOVERFLOW";\n        doRedirect = false;\n    }\n\n    function bindEvent(el, eventName, eventHandler) {\n        if (el.addEventListener){\n            el.addEventListener(eventName, eventHandler, false);\n        } else if (el.attachEvent){\n            el.attachEvent(eventName, eventHandler);\n        }\n    }\n</script>\n\n'

I just want to get the HTML of the original website.

Answer 1

You have to send User-agent along request headers to make the website to believe that the request is coming from a real web browser. So if you want the content of non-redirected url your code should be

from requests import get
content = get('http://www.cell.com/cell-stem-cell/home', headers = {'User-agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36
 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'},allow_redirects = False).content
print content

The output will be:

The URL has moved <a href="https://secure.jbs.elsevierhealth.com/action/getShar
edSiteSession?redirect=http%3A%2F%2Fwww.cell.com%2Fcell-stem-cell%2Fhome&rc=0&co
de=cell-site">here</a>

If you want the content of the redirected url then allows redirect, but include user-agent header. This method works for most of the websites that don't use dynamic content on their website. If you want to crawl data from a dynamic content website then you have to use web browser simulators like selinium.

Answer 2

you just need little work around to get that directly. Server sends a Location header when redirection is required. you just need to access the URL in that Location header.

r = requests.get('http://www.cell.com/cell-stem-cell/home')
if r.status_code==302:
    r1 = requests.get(r.headers['Location'])

you will have your required data in r1.content or r1.text