Trying to process a very simple html5 script and render it using html5lib
import html5lib
html = '''<!DOCTYPE html>
<html lang="en">
<head>
<title>Hi</title>
</head>
<body>
<script src="a.js"></script>
<script src="b.js"></script>
</body>
</html>
'''
parser = html5lib.HTMLParser(tree = html5lib.treebuilders.getTreeBuilder("lxml"))
walker = html5lib.treewalkers.getTreeWalker("lxml")
serializer = html5lib.serializer.htmlserializer.HTMLSerializer()
document = parser.parse(html)
stream = walker(document)
theHTML = serializer.render(stream)
print theHTML
The output looks like:
<!DOCTYPE html><html lang=en><head>
<title>Hi</title>
</head>
<body>
<script src=a.js></script>
<script src=b.js></script>
Yup. It just cuts off mid way. Changing the tree builder from lxml to dom does nothing. Tweaking the HTML changes the output but it's still pretty corrupt.
So the key seems to be
omit_optional_tags=False
somehow with that missing it eats the end of the output.