The spider (scrapy) does not restart

31 views Asked by At

I am trying to restart my spider once every 30 minutes, for parsing a website, to which I get an error. The first launch is successful, the next one is an error. Spider Code:

def run_spider():
    process1 = CrawlerProcess(get_project_settings())

    process1.crawl(SpiderCardSpider)

    process1.start()

    time.sleep(18)


if __name__ == '__main__':
    while True:
        run_spider()

Error message:

    raise error.ReactorNotRestartable()
twisted.internet.error.ReactorNotRestartable

I tried to configure thread and much more, at the moment I found the following solution:

def run_script_per():
    command = 'cd.. && cd .. && cd card_parc && scrapy crawl spider_card -O results.json'
     subprocess.run(command, shell=True) But I don't think it's good
0

There are 0 answers