How to get email.Header.decode_header to work with non-ASCII characters?

Question

I'm borrowing the following code to parse email headers, and additionally to add a header further down the line. Admittedly, I don't fully understand the reason for all the scaffolding around what should be straightforward usage of the email.Headers module.

Noteworthy is that Headers is not instantiated; rather its decode_header function is called:

class DecodedHeader(object):
    def __init__(self, s, folder):
        self.msg=email.message_from_string(s[1])
        self.info=parseList(s[0])
        self.folder=folder

    def __getitem__(self,name):
        if name.lower()=='folder': return self.folder
        elif name.lower()=='uid': return self.info[1][3]
        elif name.lower()=='flags': return ','.join(self.info[1][1])
        elif name.lower()=='internal-date':
            ds= self.info[1][5]
            if Options.dateFormat:
                ds= time.strftime(Options.dateFormat,imaplib.Internaldate2tuple('INTERNALDATE "'+ds+'"'))
            return ds
        elif name.lower()=='size': return self.info[1][7]
        val= self.msg.__getitem__(name)
        if val==None: return None
        return self._convert(email.Header.decode_header(val),name)
    def get(self,key,default=None):
        return self.__getitem__(key)

    def _convert(self, list, name):
        l=[]
        for s, encoding in list:
            try:    
                if (encoding!=None):
                    s=unicode(s,encoding, 'replace').encode(Options.encoding,'replace')
            except Exception, e:
                print >>sys.stderr, "Encoding error", e
            l.append(s)

        res= "".join(l)
        if Options.addr and name.lower() in ('from','to', 'cc', 'return-path','reply-to' ): res=self._modifyAddr(res)
        if Options.dateFormat and name.lower() in ('date'): res = self._formatDate(res)
        return res  

Here's the problem: When the header (val) contains non-ASCII characters such as Ä and ä, I get:

Traceback (most recent call last):
  File "v12.py", line 434, in <module>
    main()
  File "v12.py", line 396, in main
    writer.writerow(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 152, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 149, in _dict_to_list
    return [rowdict.get(key, self.restval) for key in self.fieldnames]
  File "v12.py", line 198, in get
    return self.__getitem__(key)
  File "v12.py", line 196, in __getitem__
    return self._convert(email.Header.decode_header(val),name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/email/header.py", line 76, in decode_header
    header = str(header)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 1: ordinal not in range(128)

where u'\xe4' is ä.

I've tried a few things:

• Adding # -- coding: utf-8 -- to the top of header.py
• Calling unicode() on val before passing it to decode_header()
• Calling .encode('utf-8') on val before passing it to decode_header()
• Calling .encode('ISO-8859-1') on val before passing it to decode_header()

No joy with any of the above. What is at cause here? Given that I'm looking to maintain the usage of email.Header as above (with Header not instantiated directly), how do we ensure that non-ASCII characters get successfully decoded by decode_header?

Answer 1

The header has to be encoded correctly in order to be decoded. It looks like val comes from an already existing message, so maybe that message is bad. The error indicates it is a Unicode string, but it should be a byte string at that point. The examples on in the Python help for email.header are straightforward.

Below encodes two headers that don't even use the same encoding:

>>> import email.header
>>> h = email.header.Header(u'To: Märk'.encode('iso-8859-1'),'iso-8859-1')
>>> h.append(u'From: Jòhñ'.encode('utf8'),'utf8')
>>> h
<email.header.Header instance at 0x00559F58>
>>> s = h.encode()
>>> s
'=?iso-8859-1?q?To=3A_M=E4rk?= =?utf-8?b?RnJvbTogSsOyaMOx?='

Note that the correctly encoded header is a byte string with the encoding names embedded, and it uses no non-ASCII characters.

This decodes them:

>>> email.header.decode_header(s)
[('To: M\xe4rk', 'iso-8859-1'), ('From: J\xc3\xb2h\xc3\xb1', 'utf-8')]
>>> d = email.header.decode_header(s)
>>> for s,e in d:
...  print s.decode(e)
...
To: Märk
From: Jòhñ