webp2y XML helper sanitize line breaks under python3

Question

In my web2py app I’m processing a list of items, where the user can click on a link for each item to select this. An item has an UUID, a title and a description. For a better orientation the item description is also displayed as link title. To prevent injections by and to escape tags in the description I’m using the XML sanitizer as follows:

A(this_item.title, \
  callback = URL('item', 'select', \
                 vars=dict(uuid=this_item.uuid), user_signature=True), \
  _title=XML(str_replace(this_item.description, {'\r\n':'&#13;', '<':'&#60;', '>':'&#62;'}), sanitize=True))

Using Python 2 everything was fine. Since I have switched to Python 3 I have the following problem. When the description contains line breaks the sanitizer is not working anymore. For example the following string produces by my str_replace routine is fine to be sanitized by the XML helper under Python 2 but not under Python 3:

Header

Line1
Line2
Line3

Sanitizing line breaks escaped by 
 is the problem with Python 3 (but not with Python 2). Everything else is no problem for the XML helper to sanitize (e.g. less than or greater than, I need these, since if there is no description it is generated as <no description>).

How can be line breaks sanitized by the XML helper running web2py under Python3?

Thanks for any support!

Best regards Clemens

Answer 1

This is down to a change in python's HTMLParser class between 3.4 and 3.5, where convert_charrefs started defaulting to True: Python 3.4 DeprecationWarning convert_charrefs

I think the following fix in the your web2py yatl source should correct it: https://github.com/web2py/yatl/compare/master...timnyborg:patch-1