Scrapy KeyError - Can't find spider

Question

I've got a scrapy project that will not execute with scrapy crawl <spider-name> from the command line.

I've just moved to a new development environment on ubuntu 16.04 so I wanted to double check that the issue wasn't with my set up. To do so, I created a clean virtual environment with virtualenvwrapper using python 2.7.12 and followed the tutorial instructions from the Scrapy docs. (v1.1 to mimic my other project).

Despite the new environment, I am still seeing odd behaviour from scrapy, where it won't:

• list the spiders scrapy list
• list the settings scrapy settings
• start a crawl scrapy crawl quotes

scrapy crawl quotes raises the following error:

2017-01-06 14:20:50 [scrapy] INFO: Scrapy 1.1.1 started (bot: scrapybot)
2017-01-06 14:20:50 [scrapy] INFO: Overridden settings: {}
Traceback (most recent call last):
  File "/home/alan/QueryClick/.virtualenvs/test/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 88, in _run_print_help
    func(*a, **kw)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 149, in _run_command
    cmd.run(args, opts)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 57, in run
    self.crawler_process.crawl(spname, **opts.spargs)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/crawler.py", line 162, in crawl
    crawler = self.create_crawler(crawler_or_spidercls)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/crawler.py", line 190, in create_crawler
    return self._create_crawler(crawler_or_spidercls)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/crawler.py", line 194, in _create_crawler
    spidercls = self.spider_loader.load(spidercls)
  File "/home/alan/QueryClick/.virtualenvs/test/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 43, in load
    raise KeyError("Spider not found: {}".format(spider_name))
KeyError: 'Spider not found: quotes'

My directory structure is:

└── tutorial
    ├── scrapy.cfg
    └── tutorial
        ├── __init__.py
        ├── items.py
        ├── pipelines.py
        ├── settings.py
        └── spiders
            ├── __init__.py
            └── quote_spider.py

I've also double checked all my system requirements, as specified in the Scrapy documentation. This issue is replicated across my team, on Ubuntu 14.04 following the same steps with a virtualenv and an identical setup to mine.

If anyone can shed any light on this, I would be eternally grateful.

---

Edit: Adding settings.py

The only things active in settings.py are:

BOT_NAME = 'tutorial'
SPIDER_MODULES = ['tutorial.spiders']
NEWSPIDER_MODULE = 'tutorial.spiders'
ROBOTSTXT_OBEY = True

---

Edit: Sharing scrapy.cfg

# Automatically created by: scrapy startproject
#
# For more information about the [deploy] section see:
# https://scrapyd.readthedocs.org/en/latest/deploy.html

[settings]
default = tutorial.settings

[deploy]
#url = http://localhost:6800/
project = tutorial

Answer 1

This was a conflict with some environment variables that had been set for a django project that calls the spider - they were namespaced with SCRAPY_ and must have been conflicting.

Edit: For reference: GitHub issue on undocumented environment variable(s)

Answer 2

make sure your spider is .py, if created from jupyter for example, it might be .ipynb which will not be picked up.