Scrapy cutting off long strings

211 views Asked by At

I want to extract text data from forum posts. This is my working spider:

import scrapy
import csv #not used yet


class QuotesSpider(scrapy.Spider):
    name = "quotes2"
    start_urls = [
        'https://www.motor-talk.de/faq/mercedes-e-klasse-w210-q89.html#Q3512477',
    ]

    def parse(self, response):
        xString= ' '
        xStringLink = ' '

        for i in range(4, 6): # start, stop
            xString='//*[@id="questions"]/div[2]/div['+str(i)+']/div[2]/div[1]/table/tr/td/div/text()'
            xStringLink='//*[@id="questions"]/div[2]/div['+str(i)+']/a/@name'

            scraped_info = {
                'forum post': response.xpath(xString).extract(),
                'link': response.xpath(xStringLink).extract()
            }
            yield scraped_info

However, I only get the following output: As you can see, the forum posts are cut off :( :

(...)
2017-11-19 15:52:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.motor-talk.de/faq/mercedes-e-klasse-w210-q89.html#Q3512477> (referer: None)
2017-11-19 15:52:30 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.motor-talk.de/faq/mercedes-e-klasse-w210-q89.html>
{'forum post': ['BR210, alle Modelle mit Hersteller-Schlüssel-Nr., Typ-Schlüssel-Nr., Stückzahlen CDI Common-Rail...'], 'link': ['Q3594587']}
2017-11-19 15:52:30 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.motor-talk.de/faq/mercedes-e-klasse-w210-q89.html>
{'forum post': ['Als Neujahrs-Gruß nachfolgend eine Aufstellung der Fahrgestell-Indent-Nummern, sortiert nach Prod...'], 'link': ['Q5160969']}
2017-11-19 15:52:30 [scrapy.core.engine] INFO: Closing spider (finished)
(...)

In reality, the posts are much longer, but the strings simply get cut off.

0

There are 0 answers