I'm scraping e-commerce site, and scraping multiple categories of it but some of them producing results but some of links got ERROR: Spider error processing... please help me how to sort it...

this is code

this is code 2

this is the error in console

1 Answers

0
Georgiy On

According to your error data from console error occures on urls with p argument ?p=2 - it means that your scraper doesn't process second pages of specific seacrch.
It looks like your application doesn't transfer metadata to next requests.

parse_products method of your spider require meta datafrom parse method.
I suppose that your parse_products code responsible to search results pagination looks like this:

yield Request(next_url,callback=self.parse_products)

If its true your application will be unable to read metadata and raise error like from your console.
In this case you need to add meta argument to nextpage request in your parse_products mehtod:

yield Request(next_url, meta = response.meta,callback=self.parse_products)