Best way to parse large XML document in Jython

Question

I need to parse a large (>800MB) XML file from Jython. The XML is not deeply nested, containing about a million relevant elements. I need to convert these elements into real objects.

I've used nu.xom.* successfully before, but now that I've switched from Java to Jython, the library fails with the following message:

The parser has encountered more than "64,000" entity expansions in this document; this is the limit imposed by the application.

I have not found a way to fix this, so I probably have to look for another XML library. It could be either Java or Jython-compatible Python and should be efficient. Pythonic would be great, nu.xom.* is simple but not very pythonic. Do you have any suggestions?

Answer 1

Does jython support xml.etree.ElementTree? If so, use the iterparse method to keep your memory size down. Read this and use elem.clear() as described.

Answer 2

Sax is the best way to parse large documents.

Sounds like you're hitting the default expansion limit. See this note:

https://bugs.java.com/bugdatabase/view_bug?bug_id=4843787

You need to set System property "entityExpansionLimit" to change the default.

(added) see also the answer to this question.

Answer 3

there is a lxml python library, that can parse large files, without loading data to memory. but i don't know if i jython compatible

Answer 4

Try using the SAX parser, it is great for streaming large XML files.