When I try to download a image from one url, the code works, but when I try another URL, it dosen't work. This Doesn't Work. It only creates the filename.

# This Doesn't Work.
import requests
url = 'https://ryanspressurewashing.com/wp-content/uploads/2017/06/metal-
roof-after-pressure-washing.jpg'

r = requests.get(url, stream=True)
with open('image3.jpg', 'wb') as my_file:
# Read by 4KB chunks
    for byte_chunk in r.iter_content(chunk_size=4096):
        my_file.write(byte_chunk)



#  This Works?

import requests
url = 'http://www.webscrapingfordatascience.com/files/kitten.jpg'
r = requests.get(url, stream=True)
with open('image.jpg', 'wb') as my_file:
# Read by 4KB chunks
    for byte_chunk in r.iter_content(chunk_size=4096):
        my_file.write(byte_chunk)

1 Answers

0
furas On Best Solutions

Different portals may have different security systems to block script/bots.

When you open image3.jpg in text editor you will see

<head>
<title>Not Acceptable!</title>
</head>
<body>
<h1>Not Acceptable!</h1>
<p>An appropriate representation of the requested resource could not be found on  this server. 
This error was generated by Mod_Security.</p>
</body>
</html>

Some servers may need correct headerse, cookies, session-id, etc. to give access to data.

This portal needs correct header user-agent

import requests

url = 'https://ryanspressurewashing.com/wp-content/uploads/2017/06/metal-roof-after-pressure-washing.jpg'

headers = {
  'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0'
}

r = requests.get(url, stream=True, headers=headers)

with open('image3.jpg', 'wb') as my_file:
# Read by 4KB chunks
    for byte_chunk in r.iter_content(chunk_size=4096):
        my_file.write(byte_chunk)

requests as default uses user-agent: python-requests/2.21.0 so portals can easily recognize script and block it.

You can see this header using https://httpbin.org/get

import requests

r = requests.get('https://httpbin.org/get')
print(r.text)

Result:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.21.0"
  }, 
  "origin": "83.23.39.165, 83.23.39.165", 
  "url": "https://httpbin.org/get"
}

See more functions on httpbin.org