Extract JPG urls from Bing or Google search

352 views Asked by At

Is there anyway to use Xidel to query either Bing or Google image search and then extract all the URL link for images from that search? I was interested in doing this via the command line using the Xidel.EXE. Thanks

K

1

There are 1 answers

2
MatrixView On

Sure. Great you found Xidel. Great cmdline scraper, but very few people seem to know about it. Here's a oneliner that scrapes 100 "dogs" image urls of google-images:

xidel -s "https://images.google.com" ^
      --user-agent="Mozilla/5.0 (Windows NT 6.1; WOW64;) Firefox/40" ^
      -f "form(//form,{'q':'dogs'})" ^
      -e "<div class='rg_meta'>{extract(.,'ou.:.(.+?).,',1)}</div>*"

BTW, Google actually wants you to use their API, for which you can request an APIkey, but the above command just pretends to be a browser.

Also, if you add --download at the end, it will download all pics. :-)