I am trying to restart my spider once every 30 minutes, for parsing a website, to which I get an error. The first launch is successful, the next one is an error. Spider Code:
def run_spider():
process1 = CrawlerProcess(get_project_settings())
process1.crawl(SpiderCardSpider)
process1.start()
time.sleep(18)
if __name__ == '__main__':
while True:
run_spider()
Error message:
raise error.ReactorNotRestartable()
twisted.internet.error.ReactorNotRestartable
I tried to configure thread and much more, at the moment I found the following solution:
def run_script_per():
command = 'cd.. && cd .. && cd card_parc && scrapy crawl spider_card -O results.json'
subprocess.run(command, shell=True) But I don't think it's good