Getting links from an HTML table using HTML::TableExtract and HTML::Extor in Perl

Question

My goal is to extract the links from the tables titled "Agonists," "Antagonists," and "Allosteric Regulators" in the following site:

http://www.iuphar-db.org/DATABASE/ObjectDisplayForward?objectId=1&familyId=1

I've been using HTML::TableExtract to extract the tables but have been unable to get HTML::LinkExtor to retrieve the links in question. Here is the code I have so far:

use warnings;
use strict;
use HTML::TableExtract;
use HTML::LinkExtor;

my @names = `ls /home/wallakin/LINDA/ligands/iuphar/data/html2/`; 

foreach (@names)
{
chomp ($_);

my $te = HTML::TableExtract->new( headers => [  "Ligand", 
                        "Sp.", 
                        "Action", 
                            "Affinity", 
                        "Units",
                        "Reference" ] );
my $le = HTML::LinkExtor->new();

$te->parse_file("/home/wallakin/LINDA/ligands/iuphar/data/html2/$_");

my $output = $_;
$output =~ s/\.html/\.txt/g;
open (RESET, ">/home/wallakin/LINDA/ligands/iuphar/data/links/$output") or die "Can't reset";
close RESET;
#open (DATA, ">>/home/wallakin/LINDA/ligands/iuphar/data/links/$output") or die "Can't append to file";

foreach my $ts ($te->tables)
{
    foreach my $row ($ts->rows)
    {
        $le->parse($row->[0]);
        for my $link_tag ( $le->links ) 
        {
            my %links = @$link_tag;
            print @$link_tag, "\n";
            }
        }
}
#print "Links extracted from $_\n";
}

I've tried using some sample code from another thread on this site (Perl parse links from HTML Table) to no avail. I'm not sure whether it's a problem of parsing or table recognition. Any help provided would be greatly appreciated. Thanks!

Answer 1

You don't describe what the problem is...what exactly doesn't work? What does $row->[0] contain? But part of the problem might be that TableExtract returns just the 'visible' text, not the raw html, by default. You probably want to use the keep_html option in HTML::TableExtract.

Answer 2

Try this as a base script (you only need to adapt it to fetch links) :

use warnings; use strict;
use HTML::TableExtract;
use HTML::LinkExtor;
use WWW::Mechanize;

use utf8;
binmode(STDIN, ":utf8");
binmode(STDOUT, ":utf8");
binmode(STDERR, ":utf8");

my $m = WWW::Mechanize->new( autocheck => 1, quiet => 0 );
$m->agent_alias("Linux Mozilla");
$m->cookie_jar({});

my $te = HTML::TableExtract->new(
    headers => [
        "Ligand",
        "Sp.",
        "Action",
        "Affinity",
        "Units",
        "Reference"
    ]
);

$te->parse(
    $m->get("http://tinyurl.com/jvwov9m")->content
);

foreach my $ts ($te->tables) {
    print "Table (", join(',', $ts->coords), "):\n";
    foreach my $row ($ts->rows) {
        print join(',', @$row), "\n";
    }
}