Web Scraping with Selenium: Code randomly throws StaleElementReferenceException

Question

I'm trying to scrape AliExpress for some certain items, but when the code gets to one of the items (totally non-deterministic), the urlelement in the parseItems method randomly goes stale and the method throws an exception.

The code:

package com.ardilgulez.seleniumweb;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

import java.util.List;
import java.util.concurrent.TimeUnit;

public class App {

    private static WebDriver firefoxDriver = new FirefoxDriver();

    public static boolean parseItems throws StaleElementReferenceException (List<WebElement> items){
        System.out.println(items.size());
        if(items.size() > 0){
            items.forEach((item) -> {
                WebElement urlelement = item.findElement(By.cssSelector(".detail>h3>a"));
                String href = urlelement.getAttribute("href");
                System.out.println(href);
                String title = urlelement.getAttribute("title");
                System.out.println(title);
            });
        }
        return true;
    }

    public static void main(String[] args) {
        firefoxDriver.get("https://www.aliexpress.com/");
        firefoxDriver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);

        WebElement questionElement = firefoxDriver.findElement(By.xpath("//input[@name='SearchText']"));
        questionElement.sendKeys("ESP8266");
        questionElement.submit();

        while (true) {
            try {
                (new WebDriverWait(firefoxDriver, 10))
                    .until((WebDriver webDriver) -> ((JavascriptExecutor) webDriver).executeScript("return document.readyState").equals("complete"));

                (new WebDriverWait(firefoxDriver, 10))
                    .until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//ul[@id='hs-list-items']")));

                (new WebDriverWait(firefoxDriver, 10))
                    .until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//div[@id='hs-below-list-items']")));

                System.out.println("WAIT1");

                (new WebDriverWait(firefoxDriver, 20))
                        .until((WebDriver webDriver) -> {
                            WebElement listItemsUL = (new WebDriverWait(webDriver, 10))
                                .until(ExpectedConditions.presenceOfElementLocated(By.xpath("//ul[@id='hs-list-items']")));

                            List<WebElement> items = listItemsUL.findElements(By.tagName("li"));
                            return parseItems(items);
                        });

                (new WebDriverWait(firefoxDriver, 20))
                        .until((WebDriver webDriver) -> {
                            WebElement belowListItemsDiv = (new WebDriverWait(webDriver, 10))
                                .until(ExpectedConditions.presenceOfElementLocated(By.xpath("//div[@id='hs-below-list-items']")));

                            WebElement belowListItemsUL = belowListItemsDiv.findElement(By.tagName("ul"));
                            List<WebElement> items = belowListItemsUL.findElements(By.tagName("li"));
                            return parseItems(items);
                        });

                System.out.println("WAIT2");

                WebElement nextElement = (new WebDriverWait(firefoxDriver, 10))
                    .until(ExpectedConditions.presenceOfElementLocated(By.xpath("//a[@class='page-next ui-pagination-next']")));

                System.out.println(nextElement.toString());
                System.out.println("CLICK CLICK");
                nextElement.click();

            } catch (Exception e) {
                e.printStackTrace();
                break;
            }
        }
    }
}

Sometimes the element even throws an exception after the code gets its href but before the code gets its title.

I have no idea what is going on with my code. It actually works fine until it randomly decides not to work and I have no idea why.

Answer 1

It looks like when you paginate you are not waiting next page to be properly ready, and the list may contains elements from the previous page.

To ensure that the previous page is no longer avaiable try to wait until some element from the list became stale after clicking in the pagination button, like this:

nextElement.click();
new WebDriverWait(firefoxDriver, 20)).until ExpectedConditions.stalenessOf(someElementFromTheList));