How to scrape Google Shopping with location parameters

Question

I'm trying to scrape Google Shopping using Scrapy.

start_url is https://www.google.com/search?q={}&tbm=shop

I'm injecting the keyphrases into the above URL, for example, "cement+50+kg".

But I want to run the search in a different country - for example, look up cement prices in the United States. How do I achieve this?

I was trying this out manually to see if the URL changes:

1. Changed region in Advanced Search - the search query param was added, but search results did not change accordingly.
2. Changed search country in Search Setting - the results were from the country, but I couldn't figure out the query parameter in the URL.

Also, I have only done this at a very small scale, so any heads up regarding blocking, code change, etc issues I might face with Google Shopping will be much appreciated.

Are there any major roadblocks in using Scrapy? Shall I use Selenium, instead?

Answer 1

Google uses uule parameter to pass location data. It is used to encode a place or an exact location (with latitude and longitude) into a value used in a cookie, an url or an http header.

United States == w+CAIQICINVW5pdGVkIFN0YXRlcw

You could read more about it here: https://valentin.app/uule.html

There is also a third party solution like SerpApi. It's a paid API with a free trial. We handle proxies, solve captchas, and parse all rich structured data for you.

Example python code (available in other libraries also):

from serpapi import GoogleSearch

params = {
  "engine": "google",
  "q": "cement 50 kg",
  "google_domain": "google.com",
  "gl": "us",
  "hl": "en",
  "location": "United States",
  "tbm": "shop",
  "api_key": "secret_api_key"
}

search = GoogleSearch(params)
results = search.get_dict()

Example JSON output:

"shopping_results": [
  {
    "position": 1,
    "title": "Wholesale 50kg cement bag,1 Piece",
    "link": "https://www.google.com/aclk?sa=L&ai=DChcSEwjqsIfOgpjxAhUFbW8EHfIcBvwYABAFGgJqZg&sig=AOD64_2lxrIvhKl-jSkU_uwbVdMtfbIxgA&ctype=5&q=&ved=0ahUKEwjlnYPOgpjxAhUQLKwKHeHWDl8QgeUECIcD&adurl=",
    "source": "Alibaba.com",
    "price": "$0.25",
    "extracted_price": 0.25,
    "snippet": "Trade Assurance, No Shipping Fee | Alibaba.com",
    "thumbnail": "https://serpapi.com/searches/60c7c3ad5119a69d2314814b/images/2d5510b1cc7c1a7a51a1f0d85a2f131f55c541b3ee95bfa73a7e6df83daac97a.jpeg"
  },
  {
    "position": 2,
    "title": "Riverside Plastic 94 Lb/bag Cement",
    "link": "https://www.google.com/aclk?sa=L&ai=DChcSEwjqsIfOgpjxAhUFbW8EHfIcBvwYABAEGgJqZg&sig=AOD64_1UioDhoruBK2KZZM9lpo6EC9DAxA&ctype=5&q=&ved=0ahUKEwjlnYPOgpjxAhUQLKwKHeHWDl8QgeUECJMD&adurl=",
    "source": "WhiteCap.com",
    "price": "$15.89",
    "extracted_price": 15.89,
    "snippet": "Riverside cement plastic cement, 94 lb, bag container, ASTM C1328, for making mortar, plaster and stucco. | Riverside Plastic ...",
    "thumbnail": "https://serpapi.com/searches/60c7c3ad5119a69d2314814b/images/2d5510b1cc7c1a7a51a1f0d85a2f131f981e26eb8236be3e014b2e90eb4fb4e1.jpeg"
  },
  ...
]

Check out the documentation for more details.

Disclaimer: I work at SerpApi.