How to use GoogleScraper package to scrape link from different search engines in Python

Question

I want to scrape link from different search engine for my search query in python.

For eg

Query :- "who is Sachin Tendulkar"

Output : Want link from google search , bing search.

After digging many link i found google scraper packege .

Google Scraper Link

https://pypi.python.org/pypi/GoogleScraper/0.1.37

But I didn't find any luck with this package. Can anybody help me with GoogleScraper or any alternative to scrape link

Answer 1

Hey You can achieve it by same package you mention GoogleScraper. Go through the link https://github.com/NikolaiT/GoogleScraper

Also following is the python code

from GoogleScraper import scrape_with_config, GoogleSearchError
def saveLink(self, query):
        # See in the config.cfg file for possible values
        try:
            if query:
                file_name = query.replace(" " , "_")
                self.config = {
                    'SCRAPING': {
                        'use_own_ip': 'True',
                        'keyword': query,
                        'search_engines': 'bing',
                        'num_pages_for_keyword': 1,
                        'scrape_method': 'http'
                    },
                    'SELENIUM': {
                        'sel_browser': 'chrome',
                    },
                    'OUTPUT': {
                        'output_filename': "path/" + file_name + ".json"
                    },
                    'GLOBAL': {
                        'do_caching': 'False'
                    }
                }
                
                raw_html = ""
                sqlalchemy_session = scrape_with_config(self.config)
        except Exception:
            import traceback
            print(traceback.format_exc())

If you want reuslt for more than one search engine you can add

'search_engines': 'bing, yahoo ,google',

You will get json in file output_filename