How to use scrapy and selenium to interact with the body of the scrapy response?

22 views Asked by At

I've seen a lot of people making a first request with the get method of the selenium webdriver and then scrape the response with scrapy. I want to do the opposite: a first request with scrapy and then interact with the scrapy response.body using selenium.

Avoid making the same request twice:

import scrapy
from selenium import webdriver

class MySpider(scrapy.Spider):
    name = 'my_spider'
    start_urls = ['http://example.com']

    def parse(self, response):
        self.selenium_interaction(response)

    def selenium_interaction(self, response):
        driver = webdriver.Chrome()
        driver.get(response.url)
0

There are 0 answers