In R phantomJS ran with Rselenium hangs after several iterations

466 views Asked by At

I'm using phantomJS to collect data from different sites. During data scrapping process I experience a lot of crashes when parsing sites or sites elements. Unfortunately nor phantomJS nor RSelenium don't provide any information or bag report in the console. Script just hangs without any warnings. I see that it is executing, but actually nothing happens. The only way to stop script from executing is to manually restart R. After several test I found that phantomJS usually hangs on executing remDr$findElements() commands. I tried to reran my code using firefox and RSelenium - it works normally. So the problem is in how phantomJS works.

Does anyone experience anything similar when running phantomJS? Is it possible to fix this misbehavior?

I'm using:

  1. Windows 7
  2. Selenium 2.0
  3. R version 3.1.3
  4. phantomjs-2.0.0-windows

My code:

# starting phantom server driver
phantomjsdir <- paste(mywd, "/phantomjs-2.0.0-windows/bin/phantomjs.exe", sep="" )
phantomjsUserAgent <- "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36 OPR/28.0.1750.48"
eCap <- list(phantomjs.binary.path = phantomjsdir, phantomjs.page.settings.userAgent = phantomjsUserAgent )
pJS <- phantom(pjs_cmd = phantomjsdir)
remDr <- remoteDriver(browserName = "phantomjs", extraCapabilities = eCap)
remDr$open(silent = FALSE)


mywords <- c("canon 600d", "sony 58k","nikon","nikon2","nikon 800","nikon 80","nikon 8")
timeout <- 3

#'
#' Exceuting script
#'

for (word in mywords) {

  print(paste0("searching for: ",word))
  ss.word <- word
  remDr$navigate("http://google.com")

  webElem <- remDr$findElement(using = "class", "gsfi")
  webElem$sendKeysToElement(list(enc2utf8(ss.word),key = "enter"))
  Sys.sleep(1)

  print (remDr$executeScript("return document.readyState;")[[1]])
  while (remDr$executeScript("return document.readyState;")[[1]]!= "complete" && totalwait<10) {
    Sys.sleep(timeout)
  }

  print(paste0("search completed: ",ss.word))
  elem.snippet <- remDr$findElements(using="class name",value = "rc")

  for (i in 1:length(elem.snippet)) {


    print(paste0("element opened: ",ss.word,"  pos",i))
    print(elem.snippet[[i]])
    ss.snippet.code  <- elem.snippet[[i]]$getElementAttribute('innerHTML')
    print(paste0("element element innerHTML ok"))
    elemtitle <- elem.snippet[[i]]$findChildElement(using = "class name", value = "r")
    print(paste0("element title ok"))


    elemcode <- elemtitle$getElementAttribute('innerHTML')
    print(paste0("element innerHTML ok"))


    elemtext <- elem.snippet[[i]]$findChildElement(using = "class name", value = "st")
    ss.text <- elemtext$getElementText()[[1]]
    print(paste0("element loaded: ",ss.word,"  pos",i))


    elemloc <- elem.snippet[[i]]$getElementLocation()
    elemsize <- elem.snippet[[i]]$getElementSize()
    print(paste0("element location parsed: ",ss.word,"  pos",i))

  }

  print(paste0("data collected: ",ss.word))
}
0

There are 0 answers