When I use this code, I get timeout exception after a while.

driver = webdriver.Firefox()
driver.implicitly_wait(100)


def csv_url_reader(url_obj):
    reader = csv.DictReader(url_obj, delimiter=',')
    for line in reader:
        url = line["URL"]
        driver = webdriver.Firefox()
        driver.get(url)
        try:
            title = WebDriverWait(driver, 100).until(
                EC.presence_of_element_located((By.CLASS_NAME, "some class name with title"))
            ).text
        finally:
            driver.close()
            driver.quit()
        print("Title is " + title)


if __name__ == "__main__":
    with open("url.csv") as url_obj:
        csv_url_reader(url_obj)

CSV File contains about 3 thousand links and after processing two hundredths of them it outputs an error. How can I get around this error? Can I restart the script from the last processed link?

1 Answers

1
zemiret On Best Solutions

By the trace you provided in comment, this line is throwing an exception:

title = WebDriverWait(driver, 100).until(
                EC.presence_of_element_located((By.CLASS_NAME, "some class name with title"))
            ).text

You have it in your try block but the try block does not have a corresponding except so... basically the error is not caught. Add an except block and it will work.