Flutter scrape Google Shopping

Question

I am trying to get a product price by searching for the product in Google Shopping with a given title and storeName.

I found this question on SO, which explains how you would do it with Python. How can something like this be done with Flutter?

Also like I said, is there any way to select the store?

I couldn't find anything on this. Any help is appreciated. Let me know if you need any more info!

I tried it like this:

Future<void> getGoogleShoppingResult() async {
  const String requestUrl =
      'https://www.google.com/search?q=minecraft+toys&tbm=shop';

  final response = await http.get(Uri.parse(requestUrl));
  dom.Document document = parser.parse(response.body);

  final elements = document.querySelectorAll(
    'div.sh-np__product-title.translate-content',
  );
  print(elements.first.innerHtml);
}

With this I tried to get the title of the first found product. But this never finds any product, even though I copied the selector from Google Shopping.

Answer 1

You should use these packages:

webview_flutter
http
html

And wait till JavaScript has been loaded.

This tutorial can help: https://www.youtube.com/watch?v=EHk66k___EY

The code snippet:

import 'dart:developer';

import 'package:flutter/material.dart';
import 'package:webview_flutter/webview_flutter.dart';
import 'package:html/dom.dart' as dom;

class WebViewPageExample extends StatefulWidget {
  const WebViewPageExample({Key? key}) : super(key: key);

  @override
  State<WebViewPageExample> createState() => _WebViewPageExampleState();
}

class _WebViewPageExampleState extends State<WebViewPageExample> {
  late WebViewController _controller;

  @override
  Widget build(BuildContext context) {
    return WebView(
      onWebViewCreated: _onWebViewCreated,
      javascriptMode: JavascriptMode.unrestricted,
      initialUrl: "https://www.google.com/search?q=minecraft+toys&tbm=shop",
      onPageFinished: _onPageFinished,
    );
  }

  void _onWebViewCreated(WebViewController controller) {
    _controller = controller;
  }

  Future _onPageFinished(String url) async {
    final html = await _controller.runJavascriptReturningResult(
        "new XMLSerializer().serializeToString(document);");

    final div = dom.Document.html(html)
        .querySelectorAll('div.sh-np__product-title.translate-content');

    log('Count: ${div.length}');
  }
}