Download web page using WWW::Mechanize::Firefox

Question

I'm trying to scrape a website using WWW::Mechanize::Firefox, but whenever I try to get the data it is displaying JavaScript code and the data that I need is not there. If I inspect the element on Mozilla, the data that I need is there.

Here's my current code:

#!/usr/bin/perl

use 5.010;
use strict;
use warnings;

use WWW::Mechanize::Firefox;

my $mech = WWW::Mechanize::Firefox->new();

$mech->get('link_goes_here');
$mech->allow( javascript => 0 );
$mech->content_encoding();
$mech->save_content('source.html');

Answer 1

Ok. So you have a page that builds its content using Javascript. Presumably, you have chosen to use WWW::Mechanize::Firefox instead of WWW::Mechanize because it includes support for rendering pages that are built using Javascript.

And yet, when creating your Mechanize object, you explicitly turn off the Javascript support.

$mech->allow( javascript => 0 );

I can't test this theory because you haven't told us which URL you are using, but I bet you get a better result if you change that line to:

$mech->allow( javascript => 1 );