Python 3 scrape webpage with BeautifulSoup causing UnicodeDecodeError

50 views Asked by At

I am doing a personal project. I am trying to parse a webpage. UNFORTUNATELY I did not realize that scraping a website could get your access suspended to a webpage. I made local copies of the website using hide.me but apparently it has added information that is making it difficult for BeautifulSoup to read. This is my code:

def pull_safe(location):
url = (os.getcwd())+'/HTML_SOURCES/'+location
page = open(url,encoding="ascii")
soup = BeautifulSoup(page, "html.parser", exclude_encodings=["ascii"])
hospital = list()
templist = list()
tempcount = 0
for td in soup.find('div', {'class':'report'}).parent.find_all('td'):
    if tempcount !=5:
        templist.append(td.text)
        tempcount+=1
    else:
        templist.append(td.text)
        hospital.append(templist)
        templist = list()
        tempcount = 0
return hospital

and this is the exception I am getting:

Traceback (most recent call last):


File "/home/memeputer/Documents/Projects/NYC Hospital Bed count/main.py", line 51, in <module>
    g = pull_safe(item)
  File "/home/memeputer/Documents/Projects/NYC Hospital Bed count/main.py", line 17, in pull_safe
    soup = BeautifulSoup(page, "html.parser", exclude_encodings=["utf-8"])
  File "/home/memeputer/Documents/Projects/NYC Hospital Bed count/venv/lib/python3.8/site-packages/bs4/__init__.py", line 286, in __init__
    markup = markup.read()
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 59903: invalid start byte

Any help is appreciated.

0

There are 0 answers