I'm trying to scrape some data off a site using proxies, I won't put any code here since it is a bit long and maybe not important for the question. I can just list these facts:
- The code I have does work at times, meaning it should be good
- After a couple dozen requests, the script gets blocked (even with rotating proxies!)
- Now the interesting part: after getting blocked with one proxy provider, if I immediately switch to the next one, that one is blocked too!
- I have tried with
requests.session()andhttpx.Client(), and both are failing in the same way.
So I think the only conclusion here is that my IP is somehow getting leaked by these proxies and shown to the site, triggering a block from it. Proxies in question are Brightdata Datacenter (cheap ones) and Residential proxies from proxy-cheap.com which should be a bit better.
Is there a way to prevent my IP from being sent over with proxies? I don't really think there is a different option, how else could they block rotating proxies from different providers, after letting them work initially for a while.
And by the way, I'm sure it's not some mistake by me, session is created like this
session = httpx.Client(proxies={"http://":proxy_url, "https://":proxy_url})
So that every request uses the same proxy.
I believe the easy solution would be buying an elite proxy, since some proxy will tell the server that request is made by a proxy.