I am crawling a site with 20 search results out of 200 using requests and BeautifulSoup. I only need the top 20 results, but unfortunately based on a different sorting algorithm than the default one.

The results are given as follows:

Name        ShoeSize      Length
John           8            160
Patrick        9            158
Stuart         9            170
Jack           9.5          164
...

On the page it is very easy to toggle the list by sorting by Length (which is what I want). However, the url does not change for this, so requests.get(url) does not pick it up.

The site implements is relatively easily in that in the default case it shows

<th class="sorting_asc">ShoeSize .... <th class="sorting">Length ...

while after toggling this becomes

<th class="sorting">ShoeSize .... <th class="sorting_asc">Length ...

How can I get the results with the non-default sorting (that is, on Length)?

0 Answers