Xpath won't fiind id

Question

I'm failing to get a node by its id. The code is straight forward and should be self-explaining.

#!/usr/bin/perl
use Encode; 
use utf8;
use LWP::UserAgent;   
use URI::URL; 
use Data::Dumper;
use HTML::TreeBuilder::XPath;

my $url = 'https://www.airbnb.com/rooms/1976460';
my $browser = LWP::UserAgent->new;
my $resp = $browser->get( $url, 'User-Agent' => 'Mozilla\/5.0' );

if ($resp->is_success) {
    my $base = $resp->base || '';
    print "-> base URL: $base\n";
    my $data = $resp->decoded_content;

    my $tree= HTML::TreeBuilder::XPath->new;
    $tree->parse_content( $resp->decoded_content() );
    binmode STDOUT, ":encoding(UTF-8)";
    my $price_day = $tree->find('.//*[@id="price_amount"]/');
    print Dumper($price_day);

    $tree->delete();
}

The code above prints:

-> base URL: https://www.airbnb.com/rooms/1976460
$VAR1 = undef;

How can I select a node by its ID?

Thanks in advance.

Answer 1

Take that / off the end of that XPath.

.//*[@id="price_amount"]

should do. As it is, it's not valid XPath.

Answer 2

There is a trailing slash in your XPath, that you need to remove

my $price_day = $tree->find('.//*[@id="price_amount"]');

However, from my own testing, I believe that HTML::TreeBuilder::XPath is also having trouble parsing that specific URL. Perhaps because of the conditional comments?

As an alternative approach, I would recommend using Mojo::UserAgent and Mojo::DOM instead.

The following uses the css selector div#price_amount to easily find your desired element and print it out.

use strict;
use warnings;

use Mojo::UserAgent;

my $url = 'https://www.airbnb.com/rooms/1976460';
my $dom = Mojo::UserAgent->new->get($url)->res->dom;

my $price_day = $dom->at(q{div#price_amount})->all_text;

print $price_day, "\n";

Outputs:

$285

Note, there is a helpful 8 minute introductory video to this set of modules at Mojocast Episode 5.