Is it possible to run Scrapy
on PyPy
? I've looked through the documentation and the github project, but the only place where PyPy
is mentioned is that there were some unit tests being executed on PyPy
2 years ago, see PyPy
support. There is also Scrapy fails in PyPy long discussion happened 3 years ago without a concrete resolution or a follow-up.
From what I understand, the main Scrapy's dependency Twisted
is known to work on PyPy
. Scrapy also uses lxml
for HTML parsing, which has a PyPy
-friendly fork. The other dependency, pyOpenSSL
is fully supported (thanks to @Glyph's comment).
Yes. :-)
In a bit more detail, I already had a version of pypy 2.6.0 (with pip) installed on my box. Simply running
pip install scrapy
nearly just worked for me. Turns out I needed some extra libraries for lxml. After that it was fine.Once installed, I could run the dmoz tutorial. For example:
And as requested, here's some more info on the version I'm running: