java.lang.OutOfMemoryError when processing large pgp file

Question

I want to use Camel 2.12.1 to decrypt some potentially large pgp files. The following flow results in an out of memory exception and the call stack shows that the PGPDataFormat.unmarshal() function is trying to build a ByteArray which is destined to fail if the file is large. Is there a way to pass streams around during unmarshalling?

My route:

from("file:///home/cps/camel/sftp-in?"
        + "include=.*&"  // find files using this pattern
        + "move=/home/cps/camel/sftp-archive&"  // after done adding records to queue, move file to archive
        + "delay=5000&"
        + "readLock=rename&" // readLock parameters prevent picking up file which is currently changing
        + "readLockCheckInterval=5000")
        .choice()
        .when(header(Exchange.FILE_NAME_ONLY).regex(".*pgp$|.*PGP$|.*gpg$|.*GPG$")).to("direct:decrypt")
        .otherwise()
        .to("file:///home/cps/camel/input");

from("direct:decrypt").unmarshal().pgp("file:///home/cps/.gnupg/secring.gpg", "developer", "set42now")
        .setHeader(Exchange.FILE_NAME).groovy("request.headers.get('CamelFileNameOnly').replace('.gpg', '')")
        .to("file:///home/cps/camel/input/")
        .to("log:done");

The exception which shows the converter trying to create a ByteArray:

java.lang.OutOfMemoryError: Java heap space
    at org.apache.commons.io.output.ByteArrayOutputStream.needNewBuffer(ByteArrayOutputStream.java:128)
    at org.apache.commons.io.output.ByteArrayOutputStream.write(ByteArrayOutputStream.java:158)
    at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1026)
    at org.apache.commons.io.IOUtils.copy(IOUtils.java:999)
    at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:218)
    at org.apache.camel.converter.crypto.PGPDataFormat.unmarshal(PGPDataFormat.java:238)
    at org.apache.camel.processor.UnmarshalProcessor.process(UnmarshalProcessor.java:65)

Answer 1

Try with 2.13 or 2.12-SNAPSHOT as we have improved data format and streaming recently. So likely to be better in next release.