How to scrape website that comes up with 403 error?

Question

I am trying to scrape the following web page https://jamanetwork.com/journals/jamaneurology/article-abstract/2696970 but getting an error.

url ='https://jamanetwork.com/journals/jamaneurology/article-abstract/2696970'
result = requests.get(url)
soup = BeautifulSoup(result.content, 'html.parser')
print(soup.prettify())

Result:

403 Forbidden

Request forbidden by administrative rules.

You can access the web page with no credentials, so not sure why I get 'Request forbidden' error while scraping.

Answer 1

As mentioned you should add a user-agent to your request:

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

You can check your own headers, send by the browser via opening dev tools and take a look under network section. Read more about user-agent.

Example

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}
url ='https://jamanetwork.com/journals/jamaneurology/article-abstract/2696970'
result = requests.get(url,headers=headers)
soup = BeautifulSoup(result.content, 'html.parser')
print(soup.prettify())