Crawlera & Puppeteer - Problem with Authenticantion in HTTPS

665 views Asked by At

In basic example of crawlera & puppeteer, authorization of proxy is done in this way:

await page.setExtraHTTPHeaders({
   'Proxy-Authorization': 'Basic ' + Buffer.from('<api_key>:').toString('base64'),
});

but it gives error: net::ERR_INVALID_ARGUMENT I already know that its due to chromium. In their source code I found this:

// Removing headers can't make the set of pre-existing headers unsafe, but adding headers can.

Therefor, I started using this: await page.authenticate({ username:'<api_key>',password:'' });

and it works perfectly fine with http pages. But with https it gives error: net::ERR_UNEXPECTED_PROXY_AUTH

anyone knows how to workaround that?

My versions:

  • HeadlessChrome/88.0.4298.0
  • Puppeteer: "version": "5.5.0",

Full script:

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch({
        ignoreHTTPSErrors: true,
        args: [
            '--proxy-server=proxy.crawlera.com:80',
            '--no-sandbox', 
            '--disable-setuid-sandbox'
        ]
    });
    const page = await browser.newPage();
    await page.authenticate({ username:'<apikey>',password:'' });
    
    
    
    try {
        await page.goto('https://httpbin.scrapinghub.com/get', {timeout: 180000});
    } catch(err) {
        console.log(err);
    }
  
    await browser.close();
})();
0

There are 0 answers