Python Selenium WebDriverWait() with element_to_be_clickable() does not work

Question

Here's my code:

Part 1: initialize driver

from selenium import webdriver 
from selenium.webdriver import Chrome 
from selenium.webdriver.chrome.service import Service 
from webdriver_manager.chrome import ChromeDriverManager

options = webdriver.ChromeOptions() 
options.headless = True

options.page_load_strategy = 'none' 
chrome_path = ChromeDriverManager().install() 
chrome_service = Service(chrome_path) 

driver = Chrome(options=options, service=chrome_service) 
driver.implicitly_wait(5)
driver.maximize_window()

Part 2:

# provided that url and next_button_query is initialized and correct
# provided that there is a way to check for next page

driver.get(url)
delay = 20 # seconds
while there_is_a_next_page:
   next_button = WebDriverWait(self.driver, delay).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 
   next_button_query)))
   next_button.click()

It still throws element not clickable exception. When I ask it to sleep for 1 second using time.sleep(1), it works and I was consistently able to click the button to get to the next page.

Revised Part 2:

# provided that url and next_button_query is initialized and correct
# provided that there is a way to check for next page

driver.get(url)
while there_is_a_next_page:
   time.sleep(1)
   next_button.click()

However, this solution may not be reliable and could be wasteful if there are a lot of pages. Is there any way to wait for the element to be clickable and click it right away?

Thanks in advance.

Answer 1

Apparently this is a known issue: See here, if your exception states that there is another element that would receive the click.