Some 'utf-8' codec can't decode byte

Question

I get some error when i download a website using wget

code:

import threading
import urllib.request
import os
import re
import time
import json

def wget(url):
                #self.url = url 
    data = os.popen('wget -qO- %s'% url).read()
    return data

print (wget("http://jamesholm.se/dj.php"))

Error:

Traceback (most recent call last):
  File "stand-alone-check-url.py", line 13, in <module>
    print (wget("http://jamesholm.se/dj.php"))
  File "stand-alone-check-url.py", line 10, in wget
    data = os.popen('wget -qO- %s'% url).read()
  File "/usr/local/lib/python3.4/codecs.py", line 313, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9a in position 13133: invalid start byte

How to overcome this error?

Answer 1

You can't decode arbitrary byte sequences as utf-8 encoded text:

>>> b'\xa9'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 0: invalid start byte

The page indicates that it uses utf-8 but the actual data that the server sends is not utf-8. It happens.

There is bs4.UnicodeDammit that allows you to handle data with inconsistent encodings:

import bs4 # $ pip install beautifulsoup4

print(bs4.UnicodeDammit.detwingle(b'S\x9aben  - Ostwind Rec').decode('utf-8'))
# -> Sšben  - Ostwind Rec

Answer 2

Instead of wget, use requests python module.

>>> import requests
>>> data = requests.get("http://jamesholm.se/dj.php").text
>>> print(data)