r.html.render in requests-html can not work in python 3.8

Question

I want use requests-html and Beautifulsoup to get html, but the function of'response.html.render()'didn't work.

from requests_html import HTMLSession
session = HTMLSession()
response = session.get('https://www.xxxxxxx')
response.html.render()

Answer 1

There's a bug in the requests-html library that messes up the r.html.render() method in Python 3.8. It gets stuck and won't finish loading the webpage.

To fix it, either use an older version of requests-html (0.10.0 or earlier) or switch to the arender() method instead of render(). The arender() method is a bit trickier but will work in your case!

from requests_html import AsyncHTMLSession

async def main():
    async with AsyncHTMLSession() as session:
        r = await session.get('https://example.com')
        await r.html.arender()
        data = r.html.full_text
        print(data)

if __name__ == '__main__':
    asyncio.run(main())