How to select drop-down list by Requests-HTML

16 views Asked by At

I want to scrape some dynamic website by Google Colab. Some website has drop-down list using <select> and <option> tag. I want to select each option and get rendered HTML, but HTML doesn't change. How I modify code below?

I tried this code:

from requests_html import AsyncHTMLSession

url = "https://hoge.com"

session = AsyncHTMLSession()
r = await session.get(url)
await r.html.arender(timeout=60, keep_page=True)

select = r.html.find("select#list_select_horse")
if len(select) > 0:
    print(select[0])  // <Element 'select' ... >

option = r.html.find("option")
print(option)  // [<Element 'option' value='1' selected='selected'>, <Element 'option' value='1'>, <Element 'option' value='2'>, ... ]

f = await r.html.page.select('select#list_select_horse', "3")
print(f)  // ["3"]

elements = await r.html.page.querySelectorAll('option')
print(elements)

option = r.html.find("option")
print(option)  // [<Element 'option' value='1' selected='selected'>, <Element 'option' value='1'>, <Element 'option' value='2'>, ... ]

I expect last row displays [<Element 'option' value='3' selected='selected'>, <Element 'option' value='1'>, <Element 'option' value='2'>, ... ]

0

There are 0 answers