Python Requests just got very slow, better alternative?

4.2k views Asked by At

I have a web server with some pages using Python scripts as backends. About half a dozen of them use the Requests module to fetch online resources in real time. This used to work fine for a long time. After upgrading to latest Debian (jessie), every request takes about 1 second longer.

From what I found out, the requests module in turn imports openssl, and that import is where the extra second is spent. It has been reported as a bug and will eventually be resolved, I'm certain. I don’t use any HTTPS and was hoping that I could somehow disable the SSL part, but found no suitable option/parameter.

How does this work in Debian? I was told stable releases receive only security fixes. A slow module is not security related, so will this not be resolved (the apt-get way) until next stable Debian is due?

The extra 1 second wait is not acceptable so I will likely rewrite my modules without requests. It’s a shame; I really liked requests.

What is a good replacement for requests? There seems to be three versions of urllib, not being compatible with each other?

1

There are 1 answers

0
Robert L On

Thanks to the guys in comments above, the problem was resolved. I replaced apt’s version of the Requests package with pip’s by doing something like this:

apt-get remove python-requests
apt-get install libffi-dev
pip install --upgrade requests
pip install --upgrade pyOpenSSL

The pip packages above were actually already installed, but there were later versions available. pip freeze says the versions I have now are pyOpenSSL==0.15.1 and requests==2.7.0 and it works just fine.