I am scraping amazon via Scrapy and attempting to export the price and name of the product to a csv file. When I do that, Scrapy appends the items to a list it seems and each row of the csv is a list of products for that page (the same applies to the price column). I want each item and its respective price to be printed to its own row in a CSV file. Below is my scraping code.
class ScrapeSpider(scrapy.Spider):
name = 'scrape'
start_urls = ['https://www.amazon.com/s?i=aps&k=laptop&ref=nb_sb_noss_1&url=search-alias%3Daps']
def parse(self, response):
item = AmazonItem()
name = '\n'.join(response.css('.a-text-normal.a-color-base.a-size-medium').css('::text').extract())
price = '\n'.join(response.css('.a-offscreen').css('::text').extract())
item['name'] = name
item['price'] = price
yield item
for next_page in response.css('.a-pagination .a-last a'):
yield response.follow(next_page, self.parse)
A picture of the resulting csv file
Below is the code run in the terminal to execute the scrape:
scrapy crawl scrape -o data.csv
Create a list of selectors containing each item and iterate through them creating a new selector called product. Then extract the data individually.