Why won't splash render this page?

Question

I am trying to scrape from a fashion website that uses Javascript, using Scrapy This is the page: https://www.thekooples.com/us_en/women/ready-to-wear/dresses.html

I have docker, and followed the instructions on splash docs to set up splash on localhost:8050.

I am able to render https://quotes.toscrape.com/js/ properly.

As I understand, that is a js page, and I disabled js, and it does look different when I do so.

However, I am unsuccessful in rendering the fashion webpage. This is what I get:

That is actually how the page looks like without js, so I know it is unsuccessful. What could be happening?

Answer 1

You can do try the following two things:

1. Increase the delay. You can also find an example script in splash server's homepage where you can wait for a particular element to appear.
2. You can download and print the HAR and see if any of the requests failed. If it failed then you may need to add some user-agent in your splash request.