Multi-threading with undetected_chromedriver and Selenium

Question

We are currently trying to do multi-threading using undetected_chromedriver and Selenium with the following specificities:

• multiprocessing library for multi-threading
• each browser is logged onto different Chrome profiles and does the same task

We discovered the use_multi_procs parameter in uc.Chrome() but we cannot make it work:

• Without using user_multi-procs=True.

In this case, our different browsers open up, but only the first one manages to do what we want, the others just open up and stay blocked on the Chrome start page.

• Using user_multi-procs=True.

In that case, none of the browsers open up and we just get the following error:

[WinError 5] Access is denied: 'C:\Users\clecl\appdata\roaming\undetected_chromedriver\undetected\chromedriver-win32'

We feel like the algorithm is trying to create several instances of chromedrivers in order to manage them separately but he cannot manage to do so. We found out that this folder and its two parents are "Read-only", even with admin permissions. Even after unticking the Read-only box, these folders automatically get back to being Read-only and none of the fixes we have seen work for us (attrib terminal commands, Firewall reset, admin permissions).

Would you guys see a way to solve our problem, or maybe completely different solutions that could work?

Here is a minimal reproductible example as requested.

You will need :

• Selenium 4.12.0 (4.13 is available but does not work with undetected_chromedriver),
• To use two Chrome profiles, here "Default" and "Profile 1" but you can change the names in the array,
• To copy this folder in the same folder as the .py file you are working on : "C:\Users\Your_user_name\AppData\Local\Google\Chrome\User Data"

from selenium import webdriver
import multiprocessing
from selenium.webdriver.common.action_chains import ActionChains
import undetected_chromedriver as uc

class Bot: 
    def __init__(self,profile: str, chrome_path: str):
        options = uc.ChromeOptions()
        options.add_argument(r"--user-data-dir="+chrome_path)
        options.add_argument(r'--profile-directory='+profile)
        self.driver = uc.Chrome(options=options) #add user_multi_procs=True here for second case

    def open_chrome(self):
        self.driver.get("https://www.google.com")

    def open_youtube(self):
        self.driver.get("https://www.youtube.com")

    def open_amazon(self):
        self.driver.get("https://www.amazon.com")

chrome_path=r"C:\Users\clecl\OneDrive\Documents\Bot\User Data"

PROFILE = [
    { "profile": "Default"},
    { "profile": "Profile 1"}
]

def run_bot(profile, chrome_path):
    ACTION_CHAIN = [
        "open_chrome",
        "open_youtube",
        "open_amazon"
    ]
    b = Bot(profile=profile, chrome_path=chrome_path)
    for action in ACTION_CHAIN:
        exec = getattr(b, action)
        exec()

if __name__ == "__main__":
    processes = []
    for p in PROFILE:
        process = multiprocessing.Process(target=run_bot, args=(p["profile"],chrome_path))
        processes.append(process)
        process.start()

Answer 1

What worked for us and matched what we wanted is to create a user-data-dir for each profile that we use, given that once Chrome opens one, it locks it and you cannot switch between the profiles it contains.

It certainly is heavier in terms of storage but these files will only be temporary.

Then multi-threading perfectly worked.

Answer 2

Multithreaded undetected-chromedriver can be done with https://github.com/seleniumbase/SeleniumBase UC Mode.

Here's a sample run command:

pytest --uc -n4 -s

Here's a script with 4 tests, parameterized:

import pytest
from seleniumbase import BaseCase
BaseCase.main(__name__, __file__, "--uc", "-n4", "-s")

@pytest.mark.parametrize("", [[]] * 4)
def test_multi_threaded(sb):
    sb.open("https://nowsecure.nl/#relax")
    try:
        sb.assert_text("OH YEAH, you passed!", "h1", timeout=5.25)
        sb.post_message("Selenium wasn't detected!", duration=2.8)
        sb._print("\n Success! Website did not detect Selenium! ")
    except Exception:
        sb.fail('Selenium was detected! Try using: "pytest --uc"')

Expected output if not detected:

 Success! Website did not detect Selenium! 
.
 Success! Website did not detect Selenium! 
.
 Success! Website did not detect Selenium! 
.
 Success! Website did not detect Selenium! 
.

(The . appears for every passing pytest test.)