Is there a way to speed up Javadoc (takes 7 minutes)

Question

I am building a Javadoc for a module with 2,509 classes. This currently takes 7 min or 6 files per second.

I have tried

mvn -T 1C install

However javadoc only uses 1 CPU. Is there a way to use more and/or speed up?

I am using Oracle JDK 8 update 112. My dev machine has 16 cores and 128 GB of memory.

Running flight recorder I can see that there is only one thread main

For those who are interested, I've used the following options:

<plugin>
    <artifactId>maven-javadoc-plugin</artifactId>
    <configuration>
        <additionalJOptions>
            <additionalJOption>-J-XX:+UnlockCommercialFeatures</additionalJOption>
            <additionalJOption>-J-XX:+FlightRecorder</additionalJOption>
            <additionalJOption>-J-XX:StartFlightRecording=name=test,filename=/tmp/myrecording-50.jfr,dumponexit=true</additionalJOption>
            <additionalJOption>-J-XX:FlightRecorderOptions=loglevel=debug</additionalJOption>
        </additionalJOptions>
    </configuration>
</plugin>

NOTE: One workaround is to do:

-Dmaven.javadoc.skip=true

Answer 1

@lbndev is right, at least with the default Doclet (com.sun.tools.doclets.formats.html.HtmlDoclet) that is supplied with Javadoc. A look through the source confirms the single threaded implementation:

• The starting point is in the parent class AbstractDoclet.startGeneration()
• This in turn calls new ClassTree() which calls ClassTree.buildTree() which uses a for loop to iterate the list of classes, generating models of the classes
• The next step is AbstractDoclet.generateClassFiles(), again a for loop on the packages
• This then drills down to HtmlDoclet.generateClassFiles(), which iterates package.allClasses(), again in a for loop.

(Those links are to JDK 8 source. With JDK 11 the classes have moved, but the basic for loops in HtmlDoclet and AbstractDoclet are still there.)

Some sample based profiling confirmed these are the methods that are the bottleneck:

This won't be what you're hoping to hear, but this looks like no option in the current standard Javadoc for multi-threading, at least within a single Maven module.

generateClassFiles() etc would lend themselves well to a bit of multithreading, though this would probably need to be a change in the JDK. As mentioned below AbstractDoclet.isValidDoclet() even actively blocks subclassing of HtmlDoclet. Trying to reimplement some of those loops as a third party would need to pull in a lot of other code.

A scan around other Doclet implementations (e.g. javadown) only found a similar implementation style around the package and class drilldown. It's possible others on this thread will know more.

Thinking a bit more widely, there might be room for tuning around DocFileFactory. It's clearly marked up as an internal class (not even public in the package), but it does abstract the writing of the (HTML) files. It seems possible an alternative version of this could buffer the HTML in memory, or stream directly to a zip file, to improve the IO performance. But clearly this would also need to understand the risk of change in the JDK tools.

Answer 2

javadoc, and the standard doclet, are currently fundamentally single-threaded.

It is "on the radar" to improve this, primarily by generating pages in parallel, but this means retrofitting MT-safeness to various shared data structures.

Answer 3

Use doxygen instead of the regular mvn, what you are using now.

Answer 4

Running maven with -T1C will cause maven to try to build modules in parallel, so if you have a multi-module project, at best it will build each module's javadoc in parallel (if your dependency graph between modules allow it).

The javadoc process itself is single-threaded, so you won't be able to use multiple cores to generate the javadoc of one single module.

However, since you have many classes (and possibly many @link doclets or similar ?), maybe the javadoc process could benefit from extended heap. Have you looked into GC activity ? Try adding this in your configuration, see if it helps :

<additionalJOption>-J-Xms2g</additionalJOption>
<additionalJOption>-J-Xmx2g</additionalJOption>

Answer 5

You can have Maven to use multiple threads per core in all the cores.

For eg.

mvn -T 4C install # will use 4 threads per available CPU core

You can change 4 above to whatever number you want. You have a machine with lots of resources. Try 8 or 16.

Also have you tried using javadoc-no-fork ? This will ensure javadoc is not triggered second time - https://maven.apache.org/plugins/maven-javadoc-plugin/examples/javadoc-nofork.html

Answer 6

Maven customization is a way to speed up javadoc generation.

Another approach would be to change the doclet used for generating the javadoc. The maven javadoc plugin allow you to change the doclet used to generate the javadoc

https://maven.apache.org/plugins/maven-javadoc-plugin/examples/alternate-doclet.html

I did found the following commercial doclet (I'm not affiliated with them in any way) wich claims to be faster than traditional javadoc. It offers a free/trial/commercial license. If you're realy eager to speed up your javadoc build maybe it is worth to look if it's worth the price

http://www.filigris.com/docflex-javadoc

Maybe opensource alternatives exists on internet...