Not quite understanding how to preform a request of googles servers with python requests

Question

My problem just now is not being able to form the request of googles servers properly, I've tried putting in all the request headers that my browser(Chrome) uses but that doesn't seem to work. The end goal for this is to be able to specify a search term, resolution and the file type of jpg in the request and to download the image to a folder. Any suggestions would be welcome and thanks in advance

Heres my code so far:

def funRequestsDownload(searchTerm):
print("Getting image for track ", searchTerm)
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36', 'content-length': bytes(searchTerm, 'utf-8')}
queryStringParameters = {'hl': "en", "tbm": "isch", "source": "hp", "biw":1109, "bih": 475, "q": "SEARCH TERMS", "oq":"meme", "gs_l":"img.3..35i39k1j0l9.21651.21983.0.22205.10.10.0.0.0.0.131.269.2j1.3.0....0...1.1.64.img..7.3.267.0.4mTf5BYtfj8"}
payload = {'value': searchTerm}
url = 'http://www.google.co.uk'
dataDump = requests.get(url, data=payload, headers=headers, "Query String Parameters"=queryStringParameters)
temp = dataDump.content
with open('C:/Users/Jordan/Desktop/Music Program/temp.html', 'w') as file:
    file.write(str(temp))
    file.close
return(temp)
print("Downloaded image for track ", searchTerm)

Side note, I know the only thing im saving is the html of the page, this is because it is returning a bad request page and i want to look at said error.

Answer 1

Google doesn't like people using scraping to access search results. They prefer that you use their API instead.

The API they offer is called Google Custom Search. It supports searching for images. To use their API, you'll need an adsense account. Use the API key you get from that to make your API calls.

The URL you'll want to hit is

searchUrl = "https://www.googleapis.com/customsearch/v1?q=" + \
             searchTerm + "&start=" + startIndex + "&key=" + key + "&cx=" + cx + \
             "&searchType=image"

pass that through requests to get a JSON file back with your results.

Here's some further reading.

Answer 2

First of all, http://www.google.co.uk -> http://www.google.co.uk/search, it might be a reason for a bad response.

To scrape images from Google Images, you need to parse data from the page source (ctrl+u) that is located in the <scrpt> tags. Here are steps you need to take (simplified but very close to actual code below):

1. Find all <script> tags:

soup.select('script')

2. Match images data via regex from the <script> tags:

matched_images_data = ''.join(re.findall(r"AF_initDataCallback\(([^<]+)\);", str(all_script_tags)))

3. Match desired images (full res size) via regex:

matched_images_data_fix = json.dumps(matched_images_data)
matched_images_data_json = json.loads(matched_images_data_fix)

matched_google_full_resolution_images = re.findall(r"(?:'|,),\[\"(https:|http.*?)\",\d+,\d+\]",
                                                    matched_images_data_json)

4. Extract and decode them using bytes() and decode():

for fixed_full_res_image in matched_google_full_resolution_images:
    original_size_img_not_fixed = bytes(fixed_full_res_image, 'ascii').decode('unicode-escape')
    original_size_img = bytes(original_size_img_not_fixed, 'ascii').decode('unicode-escape')

5. To save images, you can use urllib.request.urlretrieve which is probably one of the easiest solutions.

Sometimes it won't download anything because the request was sent via script (bot), and if you want to parse images from Google images or other search engines, you need to pass user-agent first, and then download the image, otherwise, the request will be blocked and it will throw an error.

Pass user-agent to urllib.request and download image:

opener=urllib.request.build_opener()
opener.addheaders=[('User-Agent','Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582')]
urllib.request.install_opener(opener)

urllib.request.urlretrieve(URL, 'your_folder/image_name.jpg')

---

Code that scrapes and downloads images locally with an example in the online IDE:

import requests, lxml, re, json, urllib.request
from bs4 import BeautifulSoup

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

params = {
    "q": "cat",      # query
    "tbm": "isch",   # image results
    "hl": "en",      # language
    "ijn": "0",      # batch of 100 images. "1" is another 100 images and so on.
}

html = requests.get("https://www.google.com/search", params=params, headers=headers)
soup = BeautifulSoup(html.text, 'lxml')


def get_images_data():

    print('\nGoogle Images Metadata:')
    for google_image in soup.select('.isv-r.PNCib.MSM1fd.BUooTd'):
        title = google_image.select_one('.VFACy.kGQAp.sMi44c.lNHeqe.WGvvNb')['title']
        source = google_image.select_one('.fxgdke').text
        link = google_image.select_one('.VFACy.kGQAp.sMi44c.lNHeqe.WGvvNb')['href']
        print(f'{title}\n{source}\n{link}\n')

    # this steps could be refactored to a more compact
    all_script_tags = soup.select('script')

    # # https://regex101.com/r/48UZhY/4
    matched_images_data = ''.join(re.findall(r"AF_initDataCallback\(([^<]+)\);", str(all_script_tags)))
    
    # https://kodlogs.com/34776/json-decoder-jsondecodeerror-expecting-property-name-enclosed-in-double-quotes
    # if you try to json.loads() without json.dumps() it will throw an error:
    # "Expecting property name enclosed in double quotes"
    matched_images_data_fix = json.dumps(matched_images_data)
    matched_images_data_json = json.loads(matched_images_data_fix)

    # https://regex101.com/r/pdZOnW/3
    matched_google_image_data = re.findall(r'\[\"GRID_STATE0\",null,\[\[1,\[0,\".*?\",(.*),\"All\",', matched_images_data_json)

    # https://regex101.com/r/NnRg27/1
    matched_google_images_thumbnails = ', '.join(
        re.findall(r'\[\"(https\:\/\/encrypted-tbn0\.gstatic\.com\/images\?.*?)\",\d+,\d+\]',
                   str(matched_google_image_data))).split(', ')

    print('Google Image Thumbnails:')  # in order
    for fixed_google_image_thumbnail in matched_google_images_thumbnails:
        # https://stackoverflow.com/a/4004439/15164646 comment by Frédéric Hamidi
        google_image_thumbnail_not_fixed = bytes(fixed_google_image_thumbnail, 'ascii').decode('unicode-escape')

        # after first decoding, Unicode characters are still present. After the second iteration, they were decoded.
        google_image_thumbnail = bytes(google_image_thumbnail_not_fixed, 'ascii').decode('unicode-escape')
        print(google_image_thumbnail)

    # removing previously matched thumbnails for easier full resolution image matches.
    removed_matched_google_images_thumbnails = re.sub(
        r'\[\"(https\:\/\/encrypted-tbn0\.gstatic\.com\/images\?.*?)\",\d+,\d+\]', '', str(matched_google_image_data))

    # https://regex101.com/r/fXjfb1/4
    # https://stackoverflow.com/a/19821774/15164646
    matched_google_full_resolution_images = re.findall(r"(?:'|,),\[\"(https:|http.*?)\",\d+,\d+\]",
                                                       removed_matched_google_images_thumbnails)


    print('\nFull Resolution Images:')  # in order
    for index, fixed_full_res_image in enumerate(matched_google_full_resolution_images):
        # https://stackoverflow.com/a/4004439/15164646 comment by Frédéric Hamidi
        original_size_img_not_fixed = bytes(fixed_full_res_image, 'ascii').decode('unicode-escape')
        original_size_img = bytes(original_size_img_not_fixed, 'ascii').decode('unicode-escape')
        print(original_size_img)

        # ------------------------------------------------
        # Download original images

        # print(f'Downloading {index} image...')
        
      # opener=urllib.request.build_opener()
      # opener.addheaders=[('User-Agent','Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582')]
      # urllib.request.install_opener(opener)

      # urllib.request.urlretrieve(original_size_img, f'Bs4_Images/original_size_img_{index}.jpg')

---

Alternatively, you can achieve the same thing by using Google Images API from SerpApi. It's a paid API with a free plan.

The difference is that you don't have to deal with regex, bypass blocks from Google, and maintain the code over time if something crashes (will be changed in the HTML). Instead, you only need to iterate over structured JSON and get the data you want.

Example code to integrate:

import os, json # json for pretty output
from serpapi import GoogleSearch

def get_google_images():
    params = {
      "api_key": os.getenv("API_KEY"),
      "engine": "google",
      "q": "pexels cat",
      "tbm": "isch"
    }

    search = GoogleSearch(params)
    results = search.get_dict()

    print(json.dumps(results['images_results'], indent=2, ensure_ascii=False))


get_google_images()

---------------
'''
[
... # other images 
  {
    "position": 100, # img number
    "thumbnail": "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRR1FCGhFsr_qZoxPvQBDjVn17e_8bA5PB8mg&usqp=CAU",
    "source": "pexels.com",
    "title": "Close-up of Cat · Free Stock Photo",
    "link": "https://www.pexels.com/photo/close-up-of-cat-320014/",
    "original": "https://images.pexels.com/photos/2612982/pexels-photo-2612982.jpeg?auto=compress&cs=tinysrgb&dpr=1&w=500",
    "is_product": false
  }
]
'''

P.S - I wrote a more in-depth blog post about how to scrape Google Images, and how to reduce the chance of being blocked while web scraping search engines.

Disclaimer, I work for SerpApi.