No Scrapy console output but print()'s are shown in console

249 views Asked by At

I am currently having a problem with a Scrapy project I am working on. I am running multiple Scrapy spiders on an AWS EC2 with Amazon Linux which I am calling from my script Main.py. The code for Main.py is as follows:

from XYZSpider.XYZ.spiders.XYZscout import XYZSpider
from MainSpider.ABC.spiders.ABCscout import ABCSpider
from twisted.internet import defer, reactor

spider_settings = [{
    'BOT_NAME' : 'ABC',
    'SPIDER_MODULES' : ['MainSpider.ABC.spiders'],
    'NEWSPIDER_MODULE' : 'MainSpider.ABC.spiders',
    'FEED_URI':'REv01/MainSpider/ABC.csv',
    'FEED_FORMAT' : 'csv',
    'LOG_LEVEL' : 'DEBUG'
    },{
    'BOT_NAME' : 'XYZ',
    'SPIDER_MODULES' : ['XYZSpider.XYZ.spiders'],
    'NEWSPIDER_MODULE' : 'XYZSpider.XYZ.spiders',
    'FEED_URI' : 'REv01/XYZSpider/XYZ.csv',
    'FEED_FORMAT' : 'csv',
    'LOG_LEVEL' : 'DEBUG',
    }]

spiders = [ABCSpider, XYZSpider]

process_1 = CrawlerRunner(spider_settings[0])
process_2 = CrawlerRunner(spider_settings[1])

@defer.inlineCallbacks
def crawl():
    yield process_1.crawl(spiders[0])   
    yield process_2.crawl(spiders[1])
    reactor.stop()
crawl()
reactor.run()    

However, I am not receiving any LOG-INFO at all from these Scrapy spiders which is odd, because when I print("TEST") from one of my spiders, it gets printed to the console. I suspect this could be something in connection with twisted.internet but I can't seem to locate the issue. Thanks in advance for the help.

BR

0

There are 0 answers