Website page buffers but will not load with Selenium chromedriver

Question

Im new to python and trying to write a program to automate the task of getting data from a NYC government website to make my daily job a little easier since i do this repetitively once a week or so. With selenium chromedriver, the page is opened and the search information is entered but then the page just buffers and buffers until i receive the error message

selenium.common.exceptions.TimeoutException: Message: timeout: Timed out receiving message from renderer: 300.000

When i try this task without chromedriver and do it myself in chrome, the page loads immediately with no problem. I don't understand why the page wont load with chromedriver?

Could it be that the website detects selenium and doesn't allow the request to go through even though i'm not denied access to the site?

Any help would be greatly appreciated!

So far I have tried:

• installing the undetected_chromedriver
• manually enabling headless=true but also, not sure if it is correct
• Setting page load strategy to eager
• downgrading to python 4.12.0

None of these things helped and the page still just buffers until receiving that timeout message

This is my code:

import setuptools
import selenium
from selenium import webdriver
import undetected_chromedriver as uc
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_experimental_option("excludeSwitches", ['enable-automation']);


if __name__ == '__main__':
    driver = uc.Chrome(headless=True,use_subprocess=False)
    options = Options()
    options.add_experimental_option("detach", True)
    options = Options()
    options.page_load_strategy = 'eager'
    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

    driver.get("https://a810-bisweb.nyc.gov/bisweb/PropertyProfileOverviewServlet?/")
    driver.maximize_window()
    boro2Select = Select(driver.find_element("id", "boro2"))
    boro2Select.select_by_visible_text('Queens')
    search_box = driver.find_element("name", "block").send_keys('2433')
    search_box = driver.find_element("name", "lot").send_keys('27')
    goButton = driver.find_element("name", "go3")
    goButton.click()

Im using sublime text and am on a mac if that helps chrome Version 118.0.5993.70 (Official Build) (x86_64) selenium version 4.12.0

Answer 1

Try to start with a more simple script (copy pasted your script after visiting site).

from selenium import webdriver
from selenium.webdriver.support.ui import Select

if __name__ == '__main__':
    driver = webdriver.Chrome()

    driver.get("https://a810-bisweb.nyc.gov/bisweb/PropertyProfileOverviewServlet?/")

    driver.maximize_window()
    boro2Select = Select(driver.find_element("id", "boro2"))
    boro2Select.select_by_visible_text('Queens')
    search_box = driver.find_element("name", "block").send_keys('2433')
    search_box = driver.find_element("name", "lot").send_keys('27')
    goButton = driver.find_element("name", "go3")
    goButton.click()

Versions I am using selenium = 4.14.0 python = 3.10