I have been trying to learn how to implement Chrome DevTools Protocol through Selenium's Bidi Webdriver in Python. I have a couple implementations working, but I am struggling with intercepting network requests.
The answer I found here has gotten me closest to achieving what I want, but I made some changes because my driver.get() was blocking on the request paused event. I'm also using send() instead of session.execute() due to previous scenarios in which I couldn't use driver.execute_cdp_cmd(), because that does not work with remote drivers. My implementation of send() was found here.
Here is my code:
def send(self, driver, cmd, params={}):
resource = "/session/%s/chromium/send_command_and_get_result" % driver.session_id
url = driver.command_executor._url + resource
body = json.dumps({'cmd': cmd, 'params': params})
response = driver.command_executor._request('POST', url, body)
return response.get('value')
async def go_to_page(self, driver):
driver.get('https://www.google.com')
async def start_listening(self, listener):
try:
async for event in listener:
logging_info(f"Event: {event.requestId}")
await trio.sleep(0.1)
@pytest.mark.trio
async def test_intercept(self):
BD = BuildDriverClass()
driver = BD.driver_setup()
async with driver.bidi_connection() as connection:
session, devtools = connection.session, connection.devtools
self.send(driver, 'Fetch.enable', {"urlPattern": '*'})
listener = session.listen(devtools.fetch.RequestPaused)
async with trio.open_nursery() as nursery:
nursery.start_soon(self.start_listening, listener)
await trio.sleep(0.2)
nursery.start_soon(self.go_to_page, driver)
It starts Chrome webdriver version=118.0.5993.70, a listener, and then opens a window and goes to google.com. The page loading symbol appears on the browser and never loads google.com. This shows me that the request was intercepted and the fetch.RequestPaused event has been triggered.
The async for loop in the start_listening task never catches the RequestPaused event and the loop never iterates. I have gotten the ResponseReceived event to work which does not involve driver.get() blocking in the process, which is why I believe my issue lies in how I am triggering the RequestPaused event. I've tried this with both a remote and local webdriver. Ideally I want this to work with a remote webdriver, but any solution would help.
I've edited this question because I discovered a partial answer to my issue. If you go to a page before starting the bidi_connection(), the listener never catches an event. My listener is now getting network.RequestWillBeSent, ResponseReceived, and other events, but I'm still having problems with the fetch.RequestPaused event. I might be lacking understanding of how I can get my driver.get() to stop blocking when the RequestPaused event triggers.