Scrapyd S3 feed export "Connection Reset by Peer"

560 views Asked by At

I'm running Scrapyd with a FEED_URI set to export to S3, but I received the following error at the very end of my scrape. Note that it successfully uploaded a few hundred kb of data to the bucket as the scrape began, then threw this error at the end:

2014-11-24 10:11:23+0000 [word] ERROR: Error storing csv feed (2285242 items) in: s3://kitchen.bucket/FoodItem.csv Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap self.__bootstrap_inner() File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/usr/local/lib/python2.7/dist-packages/scrapy/contrib/feedexport.py", line 101, in _store_in_thread key.set_contents_from_file(file) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1291, in set_contents_from_file chunked_transfer=chunked_transfer, size=size) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 748, in send_file chunked_transfer=chunked_transfer, size=size) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 949, in _send_file_internal query_args=query_args File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request retry_handler=retry_handler File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1068, in make_request retry_handler=retry_handler) File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 939, in _mexe request.body, request.headers) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 842, in sender http_conn.send(chunk) File "/usr/lib/python2.7/httplib.py", line 805, in send self.sock.sendall(data) File "/usr/lib/python2.7/ssl.py", line 329, in sendall v = self.send(data[count:]) File "/usr/lib/python2.7/ssl.py", line 298, in send v = self._sslobj.write(data) socket.error: [Errno 104] Connection reset by peer

Looks similar to boto issue 2207. I'm using gbirke's MultiFeedExporter, and received a similar error on both of my items.

0

There are 0 answers