Is there any way to scrape a search box using BeautifulSoup/requests and then search and refresh?

Question

I'm trying to make a program that can be able to make a search request on most websites such as YouTube, ESPN, My university course timetable etc...

I have looked online for various solutions but many of them point to simply adding your search query at the end of the url you are "getting", but that doesn't seem to work with all websites some of them don't update their URL's when you manually make a search, while many others might give each and every URL a unique 'id'. Would it be possible to scrape a search bar from any website and then specifying a search query and entering it? Is there a function for that?

Answer 1

You need to Use Selenium Instance to do that. You can not achieve it using BeautifulSoup or requests.

Answer 2

It's possible to use a text-based web browser and automate the search with a script. Then you can download the site you get from this browser and scrape it with BeautifulSoup or something else.