Add settings in scrapinghub spider

Question

I'm trying to enable mongodb in my spider in scrapinghub platform. For this I have to enable the extension via "EXTENSIONS" setting in the UI. But, while running the spider, I get the below error:

ValueError: Some paths in "{'scrapy.contrib.feedexport.FeedExporter': None}" convert to the same object, please update your settings

And my setting is as below:

EXTENSIONS = {'scrapy.contrib.feedexport.FeedExporter': None}

If I remove this setting, I get below error:

exceptions.ValueError: Some paths in "{'scrapy_mongodb.MongoDBPipeline': 300}" convert to the same object, please update your settings

And the setting is as below:

ITEM_PIPELINES = {'scrapy_mongodb.MongoDBPipeline': 300}

The worrying this is that both the settings work properly in local scrapyd.

Answer 1

Scrapy Cloud converts to string any given value for a setting. In your case, the dictionary you're using is interpreted as the string "{'scrapy.contrib.feedexport.FeedExporter': None}".

The problem is that there is no support for setting a string for EXTENSIONS in Scrapy (in fact, most of the settings that take dictionary values don't support it, generally because they can accept other value types so casting those strings to dict could result in something unintended).

There were some changes with Scrapy code dealing with settings so the actual issue is hidden, but that should be it. Putting your settings only in your project settings file settings.py should fix your issue.