Collecting data from web sites

Question

I have two web pages

Page 1:

<data>
<item>
<name>Item 1</name>
<url>http://someUrl.html</url>
</item>
</data>

Page 2: http://someUrl.html

<data>
<info>Info 1</info>
<info>Info 2</info>
<info>Info 3</info>
</data>

I want to crawl page 1 and follow all the links there and generate the following output

Item 1, Info 1
Item 1, Info 2
Item 1, Info 3
...

How can i achieve this using Xidel?

Answer 1

I recently found Xidel, so I'm no expert, but in my opinion it's an extremely powerful swiss-knife commandline scrape tool, that should be known by many more people.

Now, to answer your question I think the following (using html-templates) does exactly what you want:

xidel -q page1.html --extract-exclude=name -e "<name>{name:=text()}</name>*" -f "<url>{link:=text()}</url>*" -e "<info>{string-join(($name, text()), ', ')}</info>*" --hide-variable-names

Or, even shorter with CSS selectors:

xidel -q page1.html --extract-exclude=name -e "name:=css('name')" -f "link:=css('url')" -e "css('info')/string-join(($name,.),', ')" --hide-variable-names

Or, shortest with XPath:

xidel -q page1.html --extract-exclude=name -e name:=//name -f link:=//url -e "//info/string-join(($name,.),', ')" --hide-variable-names

The shortest line possible (but not in CSV format) would be:

xidel -q page1.html -e //name,//info -f //url

The above commands are for Windows, so make sure to swap the quotes <-> double quotes when on mac/ux! If you need explanation for the different parts in the lines, just ask... :-) Cheers!

Answer 2

You're talking about "all the links there", so instead of what you posted I'm going to assume as input:

<data>
  <item>
    <name>Item 1</name>
    <url>http://someUrl1.html</url>
  </item>
  <item>
    <name>Item 2</name>
    <url>http://someUrl2.html</url>
  </item>
  <item>
    <name>Item 3</name>
    <url>http://someUrl3.html</url>
  </item>
</data>

Linux:

xidel -s input.html -e 'for $item in //item for $info in doc($item/url)//info return $item/name||", "||$info'
#or
xidel -s input.html -e '
  for $item in //item
  for $info in doc($item/url)//info
  return
  $item/name||", "||$info
'

Windows:

xidel -s input.html -e "for $item in //item for $info in doc($item/url)//info return $item/name||', '||$info"
#or
xidel -s input.html -e ^"^
  for $item in //item^
  for $info in doc($item/url)//info^
  return^
  $item/name^|^|', '^|^|$info^
"

The 1^st for-loop iterates over every <item>-node. The 2^nd for-loop opens the url and iterates over every <info>-node. And the return clause is a simple string concatenation.

The output in this case:

Item 1, Info 1
Item 1, Info 2
Item 1, Info 3
Item 2, Info 4
Item 2, Info 5
Item 2, Info 6
Item 3, Info 7
Item 3, Info 8
Item 3, Info 9