wait until page.click returns true

Question

I'm scraping a website weather.com, I want to click on an html element to fetch data and show it, then I scrape that data.

But it doesn't always work for me, most of the time the click doesn't happen and I get an error trying to scrape those elements

await page.waitFor('#twc-scrollabe > table > tbody > tr:nth-child(1)')
await page.click('#twc-scrollabe > table > tbody > tr:nth-child(1)')

What should I do to ensure that the click happens?

Answer 1

Try specifying you want the element to be visible with

await page.waitFor('#twc-scrollabe > table > tbody > tr:nth-child(1)', { visible:true }) (do you notice the {visible:true} option?)

because the element could be in the page but not clickable when waitFor finds it.

More details

• when you use waitFor with a selector you're using waitForSelector under the hood (see the docs)
• waitForSelector has a series of options (see the docs)
• one of the options is visible, the docs say

wait for an element to be present in DOM and to be visible, i.e. to not have display: none or visibility: hidden CSS properties. Defaults to false.

Last but not least: @FeliFong asked you something more about your problem because you're not providing enough info about it. You could

• specify which is the page you're trying to fetch from (if it's publicly available)
• otherwise, you could make a GitHub repo, stripe out everything from your project leaving just the table you're speaking about and the Puppeteer script
• generally, when you're making a repo you find the issue yourself (it happened me lot and lot of times)
• if you don't solve the problem yourself making the repo... that's ok, we're here for that, but it's far and far easier to download a repo, launch it, fix the problem, make a pull request to you and then coming back here and giving you some explanations.
  It's useful for us but mostly for you because I tried to reply you... but I can't know if the solution I provided suite your needs... If I had the chance of checking it with a repo of your, I would have been sure I had solved the problem

[UPDATE] I fixed the issue on the repository you provided me, accept my PR on GitHub.

What I've done: I made some tests and I don't know what the hell "blocks" (or, better, stops from waiting) Puppeteer in that site... but it doesn't matter, this is my code

let i = 0;
    let found = false;
    const maxRetries = 100;
    do {
        // waits for the element we need to click
        await page.waitForSelector('#twc-scrollabe > table > tbody > tr:nth-child(1)')
        // clicks it
        await page.click('#twc-scrollabe > table > tbody > tr:nth-child(1)')
        try {
            // waits for the content we need
            await page.waitForSelector('tr:nth-child(3) > td.sunrise > div > span:nth-child(2)', {timeout:1000});
            // if the content won't be showed the code doesn't go on and the next line won't be reached
            found = true;
        } catch(e) {}
    } while(!found || i > maxRetries) // the maxRetries variable is mere prudence

• waits for the element we need to click
• clicks it
• waits for the detailed content to be showed (with a timeout of 1000 ms)
• if it won't be showed simply retries (with a maximum of 100 times)
• then go ahead after the first click, your script go ahead at a fast pace