Puppeteer issue: Same button cannot be pressed twice after page content changes (but button id did not)

Question

I'm trying to automate an checkout process using latest nodeJS and puppeteer. Everything is working except the last button press, which should be working from my understanding, but regardless nothing is happening.

Here is my code:

            else if (statusCode == 200) {
            let data = await page.evaluate(() => document.body.innerHTML);
            // Accepting cookies
            const cookieButton = 'button[id="privacy-layer-accept-all-button"]';
            await page.waitForSelector(cookieButton);
            await page.click(cookieButton);
            // Confirm cart data
            bench = new Date().getTime();
            let checkoutButton = 'div[data-test="checkout-continue-desktop-enabled"]';
            await page.waitForSelector(checkoutButton);
            await page.click(checkoutButton);
            await page.waitForNavigation({waitUntil:'networkidle0'})
            // Confirming checkout
            data = await page.evaluate(() => document.body.innerHTML);
            checkoutButton = 'div[data-test="checkout-continue-desktop-enabled"]';
            await page.waitForSelector(checkoutButton);
            await page.click(checkoutButton);

Code of the first Button (confirming cart data) - which is getting clicked (works):

<div><div class="BasketResumeHeader__BasketResumeHeaderWrapper-k9sckg-0 dWBNeC"><h3 color="black" font-family="default" spacing="base" font-size="lg" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN">Zusammenfassung</h3><span font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 ejSGsu">Gutscheine können im Zahlungsschritt hinzugefügt werden</span></div><div class="ResumeTable__StyledTableValues-sc-109drs2-0 hUoXfQ"><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 dHuLzW"></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div>Zwischensumme</div><span>101,99&nbsp;€</span></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 eYsMjW">Lieferkosten<div color="#918e8c" height="20" width="20" class="Icon-sc-1vrq823-0 cScGKb InfoIcon__StyledInfoIcon-sc-1vh1773-0 clmMtr"><svg width="32" height="32" viewBox="0 0 24 24"><path d="M12 2a10 10 0 1010 10A10 10 0 0012 2zm0 18a8 8 0 118-8 8 8 0 01-8 8zm1-7.5a.5.5 0 00-.5-.5h-1a.5.5 0 00-.5.5v3a.5.5 0 00.5.5h1a.5.5 0 00.5-.5zm0-4a.5.5 0 00-.5-.5h-1a.5.5 0 00-.5.5v1a.5.5 0 00.5.5h1a.5.5 0 00.5-.5z"></path></svg></div></div><div>Gratis</div></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div><h3 color="black" font-family="default" spacing="base" font-size="lg" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN">Gesamtsumme</h3><div class="mms-checkout-basket-summary__vat"><span color="grey2" font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 blKxsn">inkl. MwSt.</span></div></div><h3 color="black" font-family="default" spacing="base" font-size="lg" data-test="checkout-total" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN"><span>101,99&nbsp;€</span></h3></div><div class="BasketResume__StyledSummaryButtons-sc-1pu65vy-0 fFBBbE"><div data-test="checkout-continue-desktop-enabled" class="ContinueButton__StyledContinue-sc-17lzxkg-0"><button type="button" class="Buttonstyled__StyledButton-sc-140xkaw-1 jjXwnX">Zur Kasse gehen</button></div><a class="Linkstyled__StyledLinkAnchor-sc-1drhx1h-0 cSeNct BasketResume__StyledLink-sc-1pu65vy-1 duDsob" target="_self" href="/" color="black"><span color="black" font-size="sm" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 iMnkvI Linkstyled__Text-sc-1drhx1h-4 Uqzhg">Weiter einkaufen</span></a></div></div>

Code of the second button to confirm the checkout page (whole URL changes from /checkout to checkout/summary in the process) - does not work and doesn't get clicked sadly:

<div><div class="BasketResumeHeader__BasketResumeHeaderWrapper-k9sckg-0 dWBNeC"><h3 color="black" font-family="default" spacing="base" font-size="lg" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN">Zusammenfassung</h3><a class="Linkstyled__StyledLinkRouter-sc-1drhx1h-2 kgubGI BasketResumeHeader__StyledLink-k9sckg-1 dEwdDh" data-test="mms-router-link" href="/checkout/payment#discounts"><span class="Linkstyled__StyledIconWrapper-sc-1drhx1h-3 ctFeV"><div color="primary" class="Icon-sc-1vrq823-0 cRMyVe"><svg width="32" height="32" viewBox="0 0 24 24"><path d="M15.78 11.28a.75.75 0 01.22.53v.38a.77.77 0 01-.22.53l-5.14 5.13a.5.5 0 01-.71 0l-.71-.71a.49.49 0 010-.7L13.67 12 9.22 7.56a.5.5 0 010-.71l.71-.7a.5.5 0 01.71 0z"></path></svg></div></span><span color="grey5" font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 jjFdpe Linkstyled__Text-sc-1drhx1h-4 Uqzhg">Haben Sie einen Gutscheincode oder eine Geschenkkarte?</span></a></div><div class="ResumeTable__StyledTableValues-sc-109drs2-0 hUoXfQ"><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 dHuLzW"></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div>Zwischensumme</div><span>101,99&nbsp;€</span></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 eYsMjW">Lieferkosten<div color="#918e8c" height="20" width="20" class="Icon-sc-1vrq823-0 cScGKb InfoIcon__StyledInfoIcon-sc-1vh1773-0 clmMtr"><svg width="32" height="32" viewBox="0 0 24 24"><path d="M12 2a10 10 0 1010 10A10 10 0 0012 2zm0 18a8 8 0 118-8 8 8 0 01-8 1zm1-7.5a.5.5 0 00-.5-.5h-1a.5.5 0 00-.5.5v3a.5.5 0 00.5.5h1a.5.5 0 00.5-.5zm0-4a.5.5 0 00-.5-.5h-1a.5.5 0 00-.5.5v1a.5.5 0 00.5.5h1a.5.5 0 00.5-.5z"></path></svg></div></div><div>Gratis</div></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div><div direction="horizontal" class="Spacer__StyledSpacer-sc-1wx27iz-0 cfFOme"></div></div><div display="flex" class="FlexBox__StyledBox-sc-1vld6r2-0 emVcHd"><div><h3 color="black" font-family="default" spacing="base" font-size="lg" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN">Gesamtsumme</h3><div class="mms-checkout-basket-summary__vat"><span color="grey2" font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 blKxsn">inkl. MwSt.</span></div></div><h3 color="black" font-family="default" spacing="base" font-size="lg" data-test="checkout-total" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 eUyjYN"><span>101,99&nbsp;€</span></h3></div><div class="BasketResume__StyledSummaryButtons-sc-1pu65vy-0 fFBBbE"><div data-test="checkout-continue-desktop-enabled" class="ContinueButton__StyledContinue-sc-17lzxkg-0"><button type="button" class="Buttonstyled__StyledButton-sc-140xkaw-1 jjXwnX">Fortfahren und bezahlen</button><p font-size="xxs" class="Typostyled__StyledInfoTypo-sc-1jga2g7-0 iEQpGm ContinueButton__StyledDescription-sc-17lzxkg-1 dGihln">Mit Klick auf "Fortfahren und bezahlen" stimme ich den <a class="Linkstyled__StyledLinkAnchor-sc-1drhx1h-0 RsClv Summarystyled__StyledLink-sc-1euxzmh-10 fYjVAT" target="_blank" rel="noopener noreferrer" href="https://www.mediamarkt.de/de/legal/terms">AGB</a> zu. Ich habe die <a class="Linkstyled__StyledLinkAnchor-sc-1drhx1h-0 RsClv TransLink__StyledLink-sc-1f101w2-0 gwpXWL" target="_blank" rel="noopener noreferrer" href="https://www.mediamarkt.de/de/legal/terms#terms.withdrawal">Widerrufsbelehrung</a> und die <a class="Linkstyled__StyledLinkAnchor-sc-1drhx1h-0 RsClv TransLink__StyledLink-sc-1f101w2-0 gwpXWL" target="_blank" rel="noopener noreferrer" href="https://www.mediamarkt.de/de/shop/datenschutzhinweis_shop.html">Datenschutzerklärung</a> zur Kenntnis genommen.</p></div></div></div>

Sadly I can't give you better examples/hyperlinks other than those code snippets as the sites won't load without a working cookie on the site.

As you can see the selector should select the button (as it has exactly the same name on both pages), but somehow it struggles to press the button after the page gets changed after the first button was pressed successfully. I tried to add await page.waitForNavigation();, { waitUntil: 'networkidle0'} or await page.waitForTimeout(1000); because I figured maybe the data did not load in yet, but it did not really help and I do not seem to understand the problem why it isn't working.

The error message I do get is:

(node:16524) UnhandledPromiseRejectionWarning: Error: Node is either not visible or not an HTMLElement

Would appreciate if you could help me out with this problem or at least pinpoint me in the right direction how to further debug and solve this problem.

Answer 1

I have a few suggestions. Add this in to ensure that your viewing window is a good size:

await page.setViewport({width: 1024, height: 1600});

You can try clicking the button directly within the web browser context using evaluate:

await page.evaluate(selector=>{
  document.querySelector(selector).click();
},checkoutButton);

You may need to scroll to the element:

await page.evaluate(selector=>{
  elem = document.querySelector(selector);
  elem.scrollIntoView({ block: "center", inline: "center" });
  elem.click();
},checkoutButton);

As a debug/test tool, take a screenshot to see if the page after loading looks like what you expect:

await page.screenshot({path: 'test.png'});

Answer 2

From the puppeteer documentation, they recommend that you combine the click and waitForNavigation promises into a single promise and then wait for them both to resolve, this is because if you did each one of them separately, it may get stuck because waitForNavigation may get finished before click is resolved, for this reason, change this:

await page.click(checkoutButton);
await page.waitForNavigation({waitUntil:'networkidle0'})
            

To this:

await Promise.all([
page.click(checkoutButton),
page.waitForNavigation({waitUntil:'networkidle0'})
])