Scrapy - CSS selector issue

Asked by At

I would like to get the link located in a href attribute from a aelement. The url is:

I'm searching for the href of this element:

<a class="car_owner_section" href="/users/2643273" rel="nofollow"></a>

When I enter response.css('a.car_owner_section::attr(href)').get() in the terminal I just get nothing but the element exists even when I inspect view(response).

Anybody has a clue about this issue ?

1 Answers

Muhika Thomas On Best Solutions

The site seems to load on JavaScript, using splash works perfect.

Here is the code:

import scrapy
from scrapy_splash import SplashRequest

class ScrapyOverflow1(scrapy.Spider):
    name = "overflow1"

    def start_requests(self):
        url = ''

        yield SplashRequest(url=url, callback=self.parse, args={'wait': 5})

    def parse(self, response):
        links = response.xpath('//a[@class="car_owner_section"]/@href').extract()

To use splash install splash, scrapy splash and run sudo docker run -p 8050:8050 scrapinghub/splash before running the spider. Here is a great article on installing and running splash. article on scrapy spash... and also add midlewares to (also in the article) expected resultsThe result is as above