Why isn't find_all returning all results when I use BeautifulSoup?

79 views Asked by At

I am trying to parse the LinkedIn job search page. I want to access the data contained within <a> tag with class = job-card-search__link-wrapper js-focusable disabled ember-view. This should return all the job names of all job search results. The find_all method only returns 10 results but there are more than 10 results in total.

Here is the python code I have used to return the results in a list:

soup = BeautifulSoup(browser.page_source)
job_loc = results.find_all('a',{'class':'job-card-search__link-wrapper js-focusable disabled ember-view'}) #returns all jobs under the results section
job_name=[]
for job in job_loc:
    job_name.append(job.get_text().strip().replace('\n              \n                Promoted',""))
print(job_name)

Here is one example of the HTML code on the LinkedIn page that contains the job name.:

<a data-control-id="QyH5kElpSqqo8A7osUD+gg==" href="/jobs/view/1776859827/?eBP=CwEAAAFxjTMFYu0O-NJ50iRn84irOgoKIm_5169VkEgel8TqEDHartYAQgujawNHVilEARzzgM-9kSMZ5WEWapR1_XzxkH-J8iFmA1m96Q5UjVtBx-NdvkQGUrDlhsfs1vwiYGPtiiOishZ5NUot4TWaMGguIZVpPPdIzdbhIlSdl2El2dva9S8-66ZJTOaJsodHLwfHmmHHt8SNuKKzlJYEslmZN_zR7UnkQ0IwSQQc9xX4_xBFBlaAZ9Zt6u12Qa2GQWia7xxCSBIT8fYBx0wYf0z4USPeX-D3OQDjGS5kB97zl3N3AK4f672c8B8SZGMmneeF32c_kI4fTu_YBYrcBCm_6rPM7gA24Wfrf5GdRhnPq-cUaZGlgSdEL-PXdDjA_vZFdFo1vpN6ZLXw&amp;recommendedFlavor=SCHOOL_RECRUIT&amp;refId=fcdb0a51-09a5-43fe-93d9-cd9680a46f11&amp;trk=d_flagship3_search_srp_jobs" id="ember1010" class="job-card-search__link-wrapper js-focusable disabled ember-view">
            Data Analyst (2 months contract)
              <span class="job-card-search__promoted-tag label-16dp ml1">
                Promoted
              </span>
          </a>

However, every job has this similar HTML code and the text should change accordingly to the job name. Hence, I would like my python code to return all job searches.

Sorry this is my first time doing this and any help will be greatly appreciated. Thank you!

0

There are 0 answers