Add in 'if' clause using beautifulSoup

Question

I'm using BeautifulSoup to scrape a company site job positions (I have permission). The below code is able to run and the output is the url for the job posting, however I want to add in a criteria which has to be true before the url is returned.

Current Code

import requests
from bs4 import BeautifulSoup 

base = "http://implementconsultinggroup.com"
url = "http://implementconsultinggroup.com/career/#/1143"

req = requests.get(url).text
soup = BeautifulSoup(req,'html.parser')
links = soup.select("a")

for link in links:
    if "career" in link.get("href") and 'COPENHAGEN' in link.text:
        res = requests.get(base + link.get("href")).text
        soup = BeautifulSoup(res,'html.parser')
        title = soup.select_one("h1.page-intro__title").get_text() if 
soup.select_one("h1.section__title") else ""
        overview = soup.select_one("p.page-intro__longDescription").get_text()
        details = soup.select_one("div.rte").get_text()
        print(title, link, details)

What I'm trying to achieve

I want to run the above code, but only for the links where the 'Level' is = Graduate do I want to actually show the output. I have written the below, but it does not work.

level = soup.find_all('dd', {'class': 'author'})
    if "Graduate" in text

The Website I'm Scraping from

http://implementconsultinggroup.com/career/#/1143

<a href="/career/management-consultants-within-supply-chain-management/" class="box-link">
                                    <h2 class="article__title--tiny" data-searchable-text="">Management consultants within supply chain management</h2>
                                    <p class="article__longDescription" data-searchable-text="">COPENHAGEN • We are looking for bright graduates with a passion for supply chain management and supply chain planning for our planning and execution excellence team.</p>
                                    <div class="styled-link styled-icon">
                                        <span class="icon icon-icon">
                                            <i class="fa fa-chevron-right"></i>
                                        </span>
                                        <span class="icon-text">View Position</span>
                                    </div>
                                </a>


<div class="small-12 medium-3 columns top-lined">
                                <dl>
                                    <dt>Position</dt>
                                    <dd class="author">Management Consultant</dd>
                                    <dt>Level</dt>
                                    <dd class="author">Graduate</dd>
                                    <dt>Expertise</dt>
                                    <dd class="author">Operations strategy, Supply chain management</dd>
                                    <dt>Location</dt>
                                    <dd class="author">Copenhagen</dd>
                                </dl>
                            </div>

Ideal output

Ideally, I would be able to run the code I have created and it would filter out the positions where the Level != Graduate.

Answer 1

Here you go:

req = requests.get(url).text
soup = BeautifulSoup(req,'html.parser')

for li in soup.find('ul', class_='list-articles list').find_all('li'):
    level = li.find_all('dd', {'class': 'author'})[1].get_text()
    if "Graduate" in level:
        links = li.select("a")
        for link in links:
            if "career" in link.get("href") and 'COPENHAGEN' in link.text:
                res = requests.get(base + link.get("href")).text
                soup = BeautifulSoup(res,'html.parser')
                title = soup.select_one("h1.page-intro__title").get_text() if soup.select_one("h1.section__title") else ""
                overview = soup.select_one("p.page-intro__longDescription").get_text()
                details = soup.select_one("div.rte").get_text()
            print(title, link, details)