EXSLT: No more DTM IDs are available

Question

I've been searching for an answer to this problem all day long. I'm creating a stylesheet for a moderate-sized XML document (~1.5MB, ~1000 elements), which is giving me a lot of trouble. It's about creating an event-time line graph for different processing plant equipments. The XML is generated through the SAP MII QueryTempalte thingy, and is in an /Rowsets/Rowset/Row-format. All this data is processed and stored in a local node-set, in a /Equipments/Equipment/Event format. This node-set is then processed into HTML, and then rendered in the browser. Now, I'm starting to run into some trouble. I can easily pull data for the past 5 days, which result in ~900 rows of data from MII, and is processed to my node-format, resulting in just under 900 rows. But the second i hit 1017 rows fetched from MII, the stylesheet will only render about half-ways, and then stops and the "No more DTM IDs are available" exception. Now, the MII server only runs JDK 1.5.x, and I've read, that that might be a problem - only thing is, I can't do anything about that. So now I'm asking here: Is there a way to optimize my code?? I've attached some links for my XSL and a sample XML.

XSL: http://pastie.org/1566517 Samlpe XML: http://pastie.org/1566522

Now, the sample XML might not yield any "fun" visual results, and won't be able to replicate the error. But if anyone could spot an obvoius optimization, I'd love to know :) I've been thinking, that it would be nice to replace/move the calculations for startOffset, endOffset, etc, but i can't figure out how.

Hope someone can help me! :)

Answer 1

Know this question is old but just hit the same question myself, for those who also find this issue looks like there is a SAP Note detailing a JVM patch which may resolve this:

Note#1604623

https://websmp130.sap-ag.de/sap(bD1lbiZjPTAwMQ==)/bc/bsp/spn/sapnotes/index2.htm?numm=1604623

One of the fixes in the note is:

Improved resource consumption for XSLT transformation. We fixed a resource leak in the XSLT processing that will result in error "DTMException: No more DTM IDs are available" for complex XSLs using (recursive) templates. The reason for the problem is, that intermediate results of the processing step are stored in DTMs and the number of DTMs is limited to 65536.

With this change, there is a new system property "com.sap.jvm.xsltc.enableResourceOptimization". If this property is set to "true", the XSLT compiler will release intermediate results and therefore will need less resources. Processing of complex XSLs should be possible. Note that this system property is not set per default.

Answer 2

A Followup

We also encountered this problem with a different application, the basis of the error is the same. As alluded to above the fundamental problem is a limitation of the xalan, Apache project for processing xslt documents. The limitation, though triggered rarely, results from exhausting the maximum number of 65535 (16 bits worth of) DTM ids. In the above application, this is avoided, apparently, by using a resource allocation efficiency feature in SAP, but this will be inapplicable to other applications such as the one we were are using.

Changing Xalan

In summary, the 65K DTM id limitation has been around for a while (ca 2003), was patched, and then lost the patch in the 2.7 branch. The current, latest version of 2.7.2 has this problem. At its most basic level, a document id is a 32-bit integer. The fix is a source code change that increases the number of bits set aside for DTM. A more thorough discussion can be viewed here. The suggested hack is to change the number of node-bits to 12 thus increasing that for DTM to 20 increasing total DTM to 1048576. The steps are

• download xalan source
• unpack and modify src/org/apache/xml/dtm/DTMManager.java by modifying the line that sets IDENT_DTM_NODE_BITS.
• build xalan. You will probably need to install ant and set the classpath appropriately. The new jar will be located in the build directory.
• replace the previous xalan jar with the one that was just built.