Perl WWW::Mechanize for HTTPS over proxy

379 views Asked by At

I am trying to scrape a website using WWW::Mechanize module. I have configured the mechanize agent with a proxy URL and setting the proxy credentials using credentials method.

Code snippet :

my $url = 'https://abcde.com';
my $proxy_username = 'abc';
my $proxy_password = 'xyz';
my $proxy_url = 'http://xx.xxx.xxx.xxx:13228';

my $mechanize_agent = new WWW::Mechanize('cookie_jar' => {}, 'noproxy' => 1, 'ssl_opts' => { 'verify_hostname' => 0 });
$mechanize_agent->credentials( $proxy_username, $proxy_password );
$mechanize_agent->proxy(['http', 'https'], $proxy_url);
$mechanize_agent->get($url) or die 'Error in get request of $url: $@';

When URL is a plain HTTP, the script fetches and gives back the result. But when I try to hit HTTPS url I get the error

establishing SSL tunnel failed: 407 Proxy Authentication Required

I credentials are valid and I can view the website using proxy URL in Mozilla browser. Also i need t avoid using call to env_proxy() function since the proxy URL is dynamic. How can I get let my script fetch HTTPS request?

All suggestions are welcome! thanks in advance.

0

There are 0 answers