I am trying to scrape AFL odds from betfair (https://www.betfair.com.au/exchange/plus/australian-rules). I am fairly new to webscraping however have managed to scrape odds from other bookies but i am having trouble with Betfair. The data I need is within a "ui-view" tag which doesn't seem to open when I use Beautiful soup to get the HTML.

I've tried unsuccessfully to use selenium when loading the page to get the odds.

from selenium import webdriver
from bs4 import BeautifulSoup
import pprint as pp

BETFAIR_URL = "https://www.betfair.com.au/exchange/plus/australian-rules"

#functions
def parse(url):
    # open url

    driver = webdriver.Chrome(
        'C:/Users/Maroz/Downloads/chromedriver_win32 (1)/chromedriver.exe')
    # opens page
    driver.get(url)

    # parses as html
    soup = BeautifulSoup(driver.page_source, 'html.parser')

    # closes same
    driver.quit()


    return soup

betfair_soup = parse(BETFAIR_URL)

pp.pprint(betfair_soup)

#edit to show that it finds nothing in the span i need which is within the ui-#tags
price = betfair_soup.find_all("span", {"class": "bet-button-price"})
pp.pprint(price)
#output is []


I expected the betfair_soup to contain the infomation within this tag ui-view however it remains closed when printed to the terminal. Won't let me post an image because this is my first post but you might be able to see a screenshot of the tags I am trying to access here. https://imgur.com/gallery/jI3MQYY

As requested here is the html I get in terminal:

<!--[if IE]>
        <script type="text/javascript">window['isIE'] = true;</script>
        <![endif]-->
<!-- Set ie10 class: http://www.impressivewebs.com/ie10-css-hacks/ -->
<!--[if !IE]><!-->
<script>
        (function () {
            var isIE10 = Function('/*@cc_on return [email protected]*/')();

            if (isIE10) {
                document.documentElement.className += ' ie10';
            }
        })();

    </script>
<!--<![endif]-->
<bf-meta-tags></bf-meta-tags>
<bf-tooltip-guide><div class="tooltip-guide-container" ng-controller="TooltipGuideController as controller"><!-- --></div></bf-tooltip-guide>
<!-- --><ui-view></ui-view> #INFO IS IN HERE
<script src="//ie2eds.cdnppb.net/resources/eds/bundle/vendor-assets-min_4146.js"></script>
<script src="//ie2eds.cdnppb.net/resources/eds/bundle/bf-eds-static-client.min_4146_.js"></script>
<script type="text/javascript">

I put a comment where the odds are located, when i view the page source the tags are also closed, so there isn't any way of me showing you what i see when i inspect element on the odds box other than the photo link i posted above^

edit: After trying suggestion to wait for ui-view to load this is the entire response, I still couldn't access the information in the span tags though. https://pastebin.com/v6JzYa6V

FINAL EDIT: Problem solved! Thank you to everyone for your suggestions and special thanks to S Ahmed for his persistence in solving this for me!

1 Answers

0
S Ahmed On Best Solutions

Looks like it takes time to load the content of the <ui-view> tag and it is loaded by javascript. Try to wait for an internal element to be present before getting the source of the page.

Try this:

def parse(url):
    driver.get(url)
    try:
        WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.ID,"main-wrapper")))

    except:
        pp.pprint("Exception")

    finally:
        soup = BeautifulSoup(driver.page_source, 'html.parser')
        return soup
        driver.quit()

You have to import the following libraries

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Edit: Try waiting for the span.bet-button-price to be present instead of the #main-wrap

WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR,"span.bet-button-price")))