Scrapy print items to csv, each in own row

233 views Asked by At

I am scraping amazon via Scrapy and attempting to export the price and name of the product to a csv file. When I do that, Scrapy appends the items to a list it seems and each row of the csv is a list of products for that page (the same applies to the price column). I want each item and its respective price to be printed to its own row in a CSV file. Below is my scraping code.

class ScrapeSpider(scrapy.Spider):
    name = 'scrape'
    start_urls = ['https://www.amazon.com/s?i=aps&k=laptop&ref=nb_sb_noss_1&url=search-alias%3Daps']

    def parse(self, response):

        item = AmazonItem()
        name = '\n'.join(response.css('.a-text-normal.a-color-base.a-size-medium').css('::text').extract())
        price = '\n'.join(response.css('.a-offscreen').css('::text').extract())

        item['name'] = name
        item['price'] = price

        yield item

        for next_page in response.css('.a-pagination .a-last a'):
            yield response.follow(next_page, self.parse)

A picture of the resulting csv file

Below is the code run in the terminal to execute the scrape:

scrapy crawl scrape -o data.csv
1

There are 1 answers

0
ThePyGuy On BEST ANSWER

Create a list of selectors containing each item and iterate through them creating a new selector called product. Then extract the data individually.


def parse(self, response):
    items = response.css('.s-result-list .sg-col-inner')
    for product in items:
        item=AmazonItem()
        item['name'] = product.css('span.a-text-normal::text').get()
        item['price'] = product.css('.a-offscreen::text').get()
        yield item
    next_page = response.css('.a-last::attr(href)').get()
    if next_page:
        yield scrapy.Request(response.urljoin(next_page), callback=self.parse)