I can't scrape the information inside that div

526 views Asked by At

the list Li should have the 116 elements but instead it has 0 elements

import requests
from bs4 import BeautifulSoup
url='https://www.profile.nl/zoekresultaten/banden?size_width=205&size_height=60&size_inch=16'
profile= requests.get(url)
if profile.ok:

    soup = BeautifulSoup(profile.text,'lxml')

    Li= (soup.body).findAll("div", attrs={"class": "row xs-ptb-1"})
    print(len(Li))
1

There are 1 answers

1
QHarr On

It is being dynamically pulled from another endpoint you can see in network tab. With a browser this additional request is used to update the page. With requests, this does not happen; so you make the request direct to that other endpoint to get the info.

import requests

r = requests.get('https://www.profile.nl/tyres_search.json?locale=nl&size_width=205&size_height=60&size_inch=16.0').json()
print(r['products'])