How to reproduce UnicodeEncodeError?

Question

I get an error in a production system, which I fail to reproduce in a development environment:

with io.open(file_name, 'wt') as fd:
    fd.write(data)

Exception:

  File "/home/.../foo.py", line 18, in foo
    fd.write(data)

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 6400: ordinal not in range(128)

I already tried to but a lot of strange characters into the variable data.

But up to now I was not able to reproduce an UnicodeEncodeError.

What needs to be in data to get an UnicodeEncodeError?

Update

python -c 'import locale; print locale.getpreferredencoding()'
UTF-8

Update2

If I call locale.getpreferredencoding() via shell and via web request, the encoding is "UTF-8".

I updated my exception handling in my code and log the getpreferredencoding() since some days. Now it happened again (up to now I am not able to force or reproduce this), and the encoding is "ANSI_X3.4-1968"!

I have no clue where this encoding gets set ....

This puts my problem into a different direction. Leaving this question useless. My problem is now: Where does the preferred encoding get altered? But this is not part of this question.

A big thank you, for all who

Answer 1

You are relying on the default encoding for the platform; when that default encoding can't support the Unicode characters you are writing to the file, you get an encoding exception.

From the io.open() documentation:

encoding is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform dependent (whatever locale.getpreferredencoding() returns), but any encoding supported by Python can be used.

For your specific situation, the default returned by locale.getpreferredencoding() is ASCII, so any Unicode character outside the ASCII range would cause this issue, U-0080 and up.

Note that the locale is taken from your environment; if it is ASCII, that typically means the locale is set to the POSIX default locale, C.

Specify the encoding explicitly:

with io.open(file_name, 'wt', encoding='utf8') as fd:
    fd.write(data)

I used UTF-8 as an example; what you pick depends entirely on your use cases and the data you are trying to write out.

Answer 2

I tried this to reproduce the error:

with open(filename, 'wt', encoding='ascii') as fd:
    fd.write('\xa0')

Traceback (most recent call last):
File "test.py", line 2, in fd.write('\xa0')
UnicodeEncodeError: 'ascii' codec can't encode character '\xa0' in position 0: ordinal not in range(128)

Answer 3

Wrap your write in a try/except and save the data into a binary file -- you'll be able to see exactly what data is giving you issues:

with io.open(file_name, 'wt') as fd:
    try:
        fd.write(data)
    except UnicodeEncodeError:
        with open('/path/to/save/error.bin', 'wb') as err:
            err.write(data)
        raise