AsyncHTMLSession: ensure proxies are used during page render

13 views Asked by At

I have the following piece of code:

def get_browser_args_with_proxy(self, proxy_json):
        browser_args = []
        proxy_str = proxy_json['proxy'] + ":" + proxy_json['port']
        browser_args.append("--no-sandbox")
        browser_args.append('--proxy-server={}'.format(proxy_str))
        if ((len(proxy_json['username']) > 0) & (len(proxy_json['password']) > 0)):
            browser_args.append('--proxy-auth={}:{}'.format(proxy_json['username'], proxy_json['password']))
        return browser_args


    async def fetch(self):
        proxy_json = self.get_proxy_json()
        render_wait = 60
        browser_args = self.get_browser_args_with_proxy(proxy_json)
        url = "https://www.example.com"
        session = AsyncHTMLSession(browser_args=browser_args)
        r = await session.get(url)
        await r.html.arender(timeout=render_wait, keep_page=True)
        cookie = await r.html.page.cookies()

Here I am trying to obtain multiple cookies from the webpage. Would like to understand the following:

  1. Will the proxy be used during page render (when r.html.arender is called)
  2. I get an err_tunnel_connection_failed exception when proxies with authentication are used. How must I pass the authentication information?
0

There are 0 answers