Getting different results when scraping google shopping

Question

Hi I'm scraping google shopping results with php (using curl and simple html dom library). Everything was ok I was getting the result of the stores in my country, then I uploaded the code to google cloud(google app engine), and the result of the scraping got different (stores that are not located in my country) as if the result was fetched from another google domine (another country)

This is the link I used https://www.google.com.sa/search?q=coffee&safe=images&tbm=shop I tried to edit the hl,gl but it did not work.

Someone said to change the http header 'accepted-language' but that also did not work

So my question is how do I get the same result when I run the code from google cloud

Answer 1

Looking through the support for Google Shopping, it seems that this is not supported anymore according to this link.

What you may be able to try is to create a different project with an App Engine in a region closer to your location. You can find the list of locations here.

Hope you find this useful.

Answer 2

You could also try to pass the uule parameter, which is the encrypted Google location parameter.

e.g w+CAIQICIaQXVzdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM (Austin,Texas,United States)

You can read more about it here: https://blog.linkody.com/seo-local/uule-2

---

Alternatively, you could use a third party solution like SerpApi. It's a paid API with a free trial.

Example php code (available in other libraries also):

require 'path/to/google_search_results';

$query = [
  "api_key" => "secret_api_key",
  "engine" => "google",
  "q" => "coffee",
  "location" => "Austin, Texas, United States",
  "google_domain" => "google.com",
  "gl" => "us",
  "hl" => "en",
  "tbm" => "shop"
];

$search = new GoogleSearch();
$results = $search->json($query);

Example JSON output:

"inline_shopping_results": [
  {
    "position": 1,
    "block_position": "top",
    "title": "Hazelnut & French Vanilla Bundle (30% Off) 12 oz Beans",
    "price": "$47.95",
    "extracted_price": 47.95,
    "link": "https://www.google.com/aclk?sa=l&ai=DChcSEwjO8tWjiabxAhUDU3IKHUrrC5cYABAEGgJxdQ&sig=AOD64_3Ui55ttgZqIqcfG24uwghjpEXZ3w&ctype=5&q=&ved=0ahUKEwjI49CjiabxAhUCVc0KHbKhD-4Qww8I9Qo&adurl=",
    "source": "lifeboostcoffee.com",
    "shipping": "Free shipping",
    "thumbnail": "https://encrypted-tbn1.gstatic.com/shopping?q=tbn:ANd9GcQNspCBgGMVst_bukmSUPgfCrhshOZOkMMFaIVsJjlcr6P5RR7QfwcGiLzBc1jiKq_LzbgwhM2ZSjqc9p3Tr4h1KjgMbTccri-v6uIXamyMCjrMONPdN0DWyg&usqp=CAE"
  },
  ...
],
"shopping_results": [
  {
    "position": 1,
    "title": "Black Rifle Coffee Company Freedom Fuel Coffee Roast 12oz. Ground4.01Ground · Dark Roast · Bag",
    "link": "https://www.google.com/aclk?sa=L&ai=DChcSEwjO8tWjiabxAhUDU3IKHUrrC5cYABA2GgJxdQ&sig=AOD64_2h8ptBTClONQYS9yOHV7RdLQjwbw&ctype=5&q=&ved=0ahUKEwjI49CjiabxAhUCVc0KHbKhD-4Qg-UECN8L&adurl=",
    "product_link": "https://google.com/shopping/product/18134019833504388210",
    "product_id": "18134019833504388210",
    "serpapi_product_api": "https://serpapi.com/search.json?device=desktop&engine=google_product&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&product_id=18134019833504388210",
    "source": "Black Rifle Coffee Company",
    "price": "$13.99",
    "extracted_price": 13.99,
    "rating": 4,
    "reviews": 1,
    "snippet": "Ground · Dark Roast · Bag",
    "thumbnail": "https://serpapi.com/searches/60cf21bbea1442598e2972d1/images/c54459381b000cca098e49139db83d19bde8b72d3bef4a5544e7fad26ff7169d.webp"
  },
  ...
]

Check out the documentation for more details.

Disclaimer: I work at SerpApi.