Web Scraping 'NoneType' object has no attribute 'find_all' error using BeautifulSoup in python3 Juypter Notebook

25 views Asked by At

I'm new to the web scraping game and currently working on a proejct to scrape the reviews of this product (https://shopee.com.my/Women-Running-Shorts-Elastic-Waist-Running-Yoga-Short-Woman-Sports-Short-Plus-size-Gym-Fitness-Shorts-Sportswear-i.52936573.21378274920?sp_atk=57449ebe-a59d-4f29-a2ce-9c7c302d874d&xptdk=57449ebe-a59d-4f29-a2ce-9c7c302d874d)

I need to get the date, name and coments however I'm facing two issues and google wans't much of a help

  1. when I call my var it returns none
divs = product_rating.find_all('div', class_= 'shopee-product-rating')

error

  1. I need the class name of the comments however the only element I can find style. I'm not sure how else to call it the element I need its class

full code

#imported libraries

# to parse html and xml files' content
from bs4 import BeautifulSoup
# to manage requests with the server 
import requests 

import time
import datetime
import smtplib

url = 'https://shopee.com.my/Women-Running-Shorts-Elastic-Waist-Running-Yoga-Short-Woman-Sports-Short-Plus-size-Gym-Fitness-Shorts-Sportswear-i.52936573.21378274920?sp_atk=57449ebe-a59d-4f29-a2ce-9c7c302d874d&xptdk=57449ebe-a59d-4f29-a2ce-9c7c302d874d'
headers = {'User-Agent': 'Mozilla/5.0'}

#to check if my request was accepted 200 means yes
response = requests.get(url, headers = headers)
response

pages = response.content

soup = BeautifulSoup(pages, 'html.parser')
soup

#issue 1
product_rating = soup.find('div', class_='shopee-product-comment-list')
print(product_rating)

divs = product_rating.find_all('div', class_= 'shopee-product-rating')


for div in divs:
    name = div.find('div', class_= 'shopee-product-rating__author-name')
    date = div.find('div', class_= 'class="shopee-product-rating__time"')
     #issue 2
    comment = div.find('div', class_= 'style' )

for the first problem I have tried using deffirent classes but all return none which causes the error in the 2nd lline. I also tried importing selenium but didn't fix the issue.

for the second one I couldn't find a solution

0

There are 0 answers