Webscrape w/ Rselenium and Rvest from dropdown box where id changes

450 views Asked by At

I am looking to scrape some NBA date from the website numberfire at: https://www.numberfire.com/nba/daily-fantasy/daily-basketball-projections

I am trying to go into a drop down box and switch the displayed data from Fanduel to Draftkings. The 1st encountered problem is that the web page does not change with the changes to the that pull down menu. I installed and am successfully running selenium to counter this. However the next problem has been that the id for this pull down menu (and the id for all pull down menus) on this site changes with each refresh. This is causing an error in R as it says there is "NoSuchElement", as it cannot lock on to the proper menu box when it goes to the page.

Is there a way with RSelenium to or another package to fix this?

Here is my code in R:

require(RSelenium)
remDr <- remoteDriver(remoteServerAddr = "192.168.99.100", port = 4445, browserName = "chrome")

remDr$open()
remDr$navigate("https://www.numberfire.com/nba/daily-fantasy/daily-basketball-projections")

iframe <- remDr$findElement(using='id', value="select2-dy8e-container")
remDr$switchToFrame(iframe)


option <- remDr$findElement(using = 'xpath', "//*/option[@value = 'DraftKings']")
option$clickElement()
option
1

There are 1 answers

0
CooperBuckeye05 On

Update after doing a lot of searching on nonstatic Id's I came up with this and it worked:

remDr <- remoteDriver(remoteServerAddr = "192.168.99.100", port = 4445, browserName = "chrome")
remDr$open()
remDr$navigate("https://www.numberfire.com/nba/daily-fantasy/daily-basketball-projections")
webElem <- remDr$findElement('xpath', '//*[(@class = "dropdown-custom dfs-option select2-hidden-accessible")]/option[@value = "4"]')
webElem$clickElement()