Chrome headless ignores system proxy

Question

Edit: Here's the a youtube video demonstrating the problem:

https://youtu.be/MIZjHKCVr1s

I am running selenium with chrome using the --headless flag but apparently it seems that using --headless ignores system proxy. I had tested this on mac before where it didn't bypass proxy but on windows 10 VM on my mac it seems to bypass the proxy.

code :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--hide-scrollbars')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(chrome_options = options)

driver.get('https://wtfismyip.com')
print(driver.find_element_by_tag_name('body').text)

If i don't use the head less flag. It show the system proxy's IP but if use it it shows my real IP.

Answer 1

There could be a number of reasons for this: If I had to guess it is Ip leakage. Low quality proxies do not mask your ip's on various websites. A good website to test Ip leakage is whoer.net. Ip address is only one factor, you must also consider browser fingerprinting and user agent spoofing. Ideally you want whoer.net to have 100% anonymity rating.

Also, ensure you have the latest Chrome and Firefox installed.

I have tested your code and it works. Perhaps providing your full code with public proxies may help others replicate your job exactly as you see it. I can only guess how you are using proxies and so my code might differ. I am also using windows

As far as I can tell, there doesn't seem to be anything wrong with the code you have provided so I am assuimg that your ip is leaking through another means. Look into browser fingerprinting and disable webrtc and look closely at whoer.net.

Lastly, Chrome and Firefox have headless support now in the beta editions. I believe this has always been the case for Linux. Mac I am not sure on this.

Alternatively, if you wanted to use headless browser try phantom Js. Though that is being replaced by what I have mentioned above. I believe there are alternative methods for headless browsing without having to install newer versions of Chrome and Firefox though I do not know how.

EDIT: Here is an example of how you could use a proxy with Chrome driver:

from selenium import webdriver

PROXY = "23.23.23.23:2323" # IP:PORT or HOST:PORT

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % PROXY)

chrome = webdriver.Chrome(chrome_options=chrome_options)
chrome.get("http://whatismyipaddress.com")

I tried this in Selenium and as far as I can tell it worked.

Here's a list of public proxies. https://free-proxy-list.net/ . Please note that quite a few of them do not work so I had to scroll through 3 until they work. I don't think it's necessary to change your network settings for proxies just use the code above.

Hope this was helpful.