BouncyCastle 'Premature end of stream in PartialInputStream' decrypting CSV

Question

I have implemented a PGP service using BouncyCastle.Cryptography v2.3.0 NuGet in C# .NET 7.0.

I have used the service for decrypting files pulled from an SFTP server successfully. Having started processing larger CSV files, I get the following exception:

Premature end of stream in PartialInputStream

This is being thrown from BcpgInputStream.cs starting at line 338:

338: public override int Read(Span<byte> buffer)
{
    do
    {
        if (dataLength != 0)
        {
            int count = buffer.Length;
            int readLen = (dataLength > count || dataLength < 0) ? count : dataLength;
            int len = m_in.Read(buffer[..readLen]);
            if (len < 1)
                throw new EndOfStreamException("Premature end of stream in PartialInputStream");

            dataLength -= len;
            return len;
        }
    }
    while (partial && ReadPartialDataLength() >= 0);

    return 0;
}

And the stacktrace:

   at Org.BouncyCastle.Bcpg.BcpgInputStream.PartialInputStream.Read(Span`1 buffer) in /_/crypto/src/bcpg/BcpgInputStream.cs:line 350
   at System.IO.BufferedStream.Read(Span`1 destination) in /_/src/libraries/System.Private.CoreLib/src/System/IO/BufferedStream.cs:line 562
   at Org.BouncyCastle.Bcpg.BcpgInputStream.Read(Span`1 buffer) in /_/crypto/src/bcpg/BcpgInputStream.cs:line 67
   at Org.BouncyCastle.Utilities.IO.Streams.CopyTo(Stream source, Stream destination, Int32 bufferSize) in /_/crypto/src/util/io/Streams.cs:line 31
   at Org.BouncyCastle.Utilities.IO.BaseInputStream.CopyTo(Stream destination, Int32 bufferSize) in /_/crypto/src/util/io/BaseInputStream.cs:line 21
   at [redacted]ExternalPGPCryptographicService.Decrypt(Byte[] abPrivateKey, Byte[] abPassword, Byte[] abEncryptedData) in [redacted]\ExternalPGPCryptographicService.cs:line 188
   at [redacted]TestHarness.FormTestHarness._btnPGPCryptoDecrypt_Click(Object sender, EventArgs e) in [redacted]TestHarness.cs:line 86

The exception is thrown when I call:

streamDecrypted.CopyTo(memoryStream) 

Any way you try to read the streamDecrypted (Streams.PipeAll, streamDecrypted.Read etc), the exception is thrown.

The source file to decrypt is a CSV. It is encrypted on Windows 11 running GnuPG v2.4.4 with the following command:

gpg --encrypt -r [email protected] '.\FILE.csv'

The file successfully decrypts using gpg CLI.

The PGP key pair is generated from within my implementation, and uses RSA 3072, with ZIP compression capability, although I can reproduce this with a PGP key pair generated via gpg cli with the --openpgp flag specified (BouncyCastle cannot handle AEAD header).

I have reproduced this using any form of data in the source file, so long as the file is larger than 1364KB (or maybe its a red herring, and the issue is unrelated?!).

The decryption will fail with the following minimum file size:

Source CSV: 1364KB

Encrypted csv.gpg: 8.44KB

If I take 1 character out of the file and re-encrypt, it will decrypt successfully.

My implementation:

public byte[] Decrypt(byte[] abPrivateKey, byte[] abPassword, byte[] abEncryptedData)
{
    PgpObjectFactory pgpObjectFactory;
    PgpEncryptedDataList pgpEncryptedDataList;
    PgpObject pgpObject;
    PgpPrivateKey pgpPrivateKey;
    PgpPublicKeyEncryptedData pgpPublicKeyEncryptedData;
    PgpSecretKeyRingBundle pgpSecretKeyRingBundle;
    PgpLiteralData pgpLiteralData;

    // invalid provate key so don't bother
    if (abPrivateKey.Length == 0)
    {
        throw new Exception("Invalid private key");
    }

    // no encrypted data so don't bother
    if (abEncryptedData.Length == 0)
    {
        return new byte[0];
    }

    // load the private key and the encrypted data into memory streams
    using (Stream streamPrivateKey = new MemoryStream(abPrivateKey))
    using (Stream streamEncryptedData = new MemoryStream(abEncryptedData))
    {
        // create a new pgp object factory using the encrypted data
        pgpObjectFactory = new PgpObjectFactory(streamEncryptedData);

        pgpObject = pgpObjectFactory.NextPgpObject();

        // try find the encrypted pgp data 
        if (pgpObject is PgpEncryptedDataList)
        {
            pgpEncryptedDataList = (PgpEncryptedDataList)pgpObject;
        }
        else
        {
            pgpEncryptedDataList = (PgpEncryptedDataList)pgpObjectFactory.NextPgpObject();
        }

        pgpPrivateKey = null;
        pgpPublicKeyEncryptedData = null;

        Stream streamPrivateKeyClear;
        PgpObjectFactory pgpObjectFactoryPrivateKey;
        PgpObject pgpMessage;

        // create a keyring bundle from the private key stream
        pgpSecretKeyRingBundle = new PgpSecretKeyRingBundle(PgpUtilities.GetDecoderStream(streamPrivateKey));

        // try find the private key 
        foreach (PgpPublicKeyEncryptedData pgpPublicKeyEncrypted in pgpEncryptedDataList.GetEncryptedDataObjects())
        {
            // this is bad, it converts to string first
            pgpPrivateKey = FindSecretKey(pgpSecretKeyRingBundle, pgpPublicKeyEncrypted.KeyId, abPassword);

            if (pgpPrivateKey != null)
            {
                pgpPublicKeyEncryptedData = pgpPublicKeyEncrypted;
                break;
            }
        }

        // we must have a private key at this point
        if (pgpPrivateKey == null)
        {
            throw new Exception("Secret key not found");
        }

        // find the public key from the private key bundle
        streamPrivateKeyClear = pgpPublicKeyEncryptedData.GetDataStream(pgpPrivateKey);

        // create private key
        pgpObjectFactoryPrivateKey = new PgpObjectFactory(streamPrivateKeyClear);

        // find the encrypted data in the pgp object
        pgpMessage = pgpObjectFactoryPrivateKey.NextPgpObject();

        // find out if the message is compressed
        if (pgpMessage is PgpCompressedData)
        {
            PgpCompressedData pgpCompressedData;
            PgpObjectFactory pgpObjectFactoryCompressedData;

            // decompress the message
            pgpCompressedData = (PgpCompressedData)pgpMessage;
            pgpObjectFactoryCompressedData = new PgpObjectFactory(pgpCompressedData.GetDataStream());

            pgpMessage = pgpObjectFactoryCompressedData.NextPgpObject();
        }

        if (pgpMessage is PgpOnePassSignatureList)
        {
            throw new Exception("The PGP message has been signed and must be verified.");
        }

        // at this point it must be literal data. i.e. the actual message
        if (pgpMessage is PgpLiteralData)
        {
            // verify the integrity of the message if it is enabled
            if (pgpPublicKeyEncryptedData.IsIntegrityProtected())
            {
                if (pgpPublicKeyEncryptedData.Verify() == false)
                {
                    throw new Exception("Failed integrity check");
                }
            }

            // now get the data from the message
            pgpLiteralData = (PgpLiteralData)pgpMessage;

            // pull out the decrypted data from the input stream
            using (MemoryStream memoryStream = new MemoryStream())
            using (Stream streamDecrypted = pgpLiteralData.GetInputStream())
            {
                streamDecrypted.CopyTo(memoryStream);

                return memoryStream.ToArray();
            }

        }

        throw new Exception("The PGP message is in an unexpected state.");
    }
}

Due to the architecture of the wider application, the file is handled as a base64 string until it reaches the PGP service, at which point I Convert.FromBase64String and pass the bytes into the Decrypt function.

Any help would be greatly appreciated.

Answer 1

Having looked through other implementations, particularly PgpCore, I found I was reading the decrypted stream too late.

public byte[] Decrypt(byte[] abPrivateKey, byte[] abPassword, byte[] abEncryptedData)
{
    PgpObjectFactory pgpObjectFactory;
    PgpEncryptedDataList pgpEncryptedDataList;
    PgpObject pgpObject;
    PgpPrivateKey pgpPrivateKey;
    PgpPublicKeyEncryptedData pgpPublicKeyEncryptedData;
    PgpSecretKeyRingBundle pgpSecretKeyRingBundle;

    // invalid provate key so don't bother
    if (abPrivateKey.Length == 0)
    {
        throw new Exception("Invalid private key");
    }

    // no encrypted data so don't bother
    if (abEncryptedData.Length == 0)
    {
        return new byte[0];
    }

    // load the private key and the encrypted data into memory streams
    using (Stream streamPrivateKey = new MemoryStream(abPrivateKey))
    using (Stream streamEncryptedData = new MemoryStream(abEncryptedData))
    using (MemoryStream streamOutput = new MemoryStream())
    using (CompositeDisposable disposables = new CompositeDisposable())
    {
        // create a new pgp object factory using the encrypted data
        pgpObjectFactory = new PgpObjectFactory(streamEncryptedData);

        pgpObject = pgpObjectFactory.NextPgpObject();

        // try find the encrypted pgp data 
        if (pgpObject is PgpEncryptedDataList)
        {
            pgpEncryptedDataList = (PgpEncryptedDataList)pgpObject;
        }
        else
        {
            pgpEncryptedDataList = (PgpEncryptedDataList)pgpObjectFactory.NextPgpObject();
        }

        pgpPrivateKey = null;
        pgpPublicKeyEncryptedData = null;

        Stream streamPrivateKeyClear;
        PgpObjectFactory pgpObjectFactoryPrivateKey;
        PgpObject pgpMessage;

        // create a keyring bundle from the private key stream
        pgpSecretKeyRingBundle = new PgpSecretKeyRingBundle(PgpUtilities.GetDecoderStream(streamPrivateKey));

        // try find the private key 
        foreach (PgpPublicKeyEncryptedData pgpPublicKeyEncrypted in pgpEncryptedDataList.GetEncryptedDataObjects())
        {
            // this is bad, it converts to string first
            pgpPrivateKey = FindSecretKey(pgpSecretKeyRingBundle, pgpPublicKeyEncrypted.KeyId, abPassword);

            if (pgpPrivateKey != null)
            {
                pgpPublicKeyEncryptedData = pgpPublicKeyEncrypted;
                break;
            }
        }

        // we must have a private key at this point
        if (pgpPrivateKey == null)
        {
            throw new Exception("Secret key not found");
        }

        // find the public key from the private key bundle
        streamPrivateKeyClear = pgpPublicKeyEncryptedData.GetDataStream(pgpPrivateKey);

        // create private key
        pgpObjectFactoryPrivateKey = new PgpObjectFactory(streamPrivateKeyClear);

        // find the encrypted data in the pgp object
        pgpMessage = pgpObjectFactoryPrivateKey.NextPgpObject();

        // find out if the message is compressed
        if (pgpMessage is PgpCompressedData)
        {
            PgpCompressedData pgpCompressedData;
            PgpObjectFactory pgpObjectFactoryCompressedData;

            // decompress the message
            pgpCompressedData = (PgpCompressedData)pgpMessage;
            pgpObjectFactoryCompressedData = new PgpObjectFactory(pgpCompressedData.GetDataStream());

            pgpMessage = pgpObjectFactoryCompressedData.NextPgpObject();

            if (pgpMessage is PgpOnePassSignatureList || pgpMessage is PgpSignatureList)
            {
                pgpMessage = pgpObjectFactoryCompressedData.NextPgpObject();
            }

            using(Stream streamDecrypted = ((PgpLiteralData)pgpMessage).GetInputStream())
            {
                streamDecrypted.CopyTo(streamOutput);

                return streamOutput.ToArray();
            }
        }

        // at this point it must be literal data. i.e. the actual message
        if (pgpMessage is PgpLiteralData)
        {
            // verify the integrity of the message if it is enabled
            if (pgpPublicKeyEncryptedData.IsIntegrityProtected() == false || pgpPublicKeyEncryptedData.Verify())
            {
                using (Stream streamDecrypt = ((PgpLiteralData)pgpMessage).GetInputStream())
                {
                    streamDecrypt.CopyTo(streamOutput);

                    // it's valid. Finished decryption
                    return streamOutput.ToArray();
                }
            }

            throw new Exception("Failed integrity check");
        }

        if (pgpMessage is PgpOnePassSignatureList)
        {
            throw new Exception("The PGP message has been signed and must be verified.");
        }

        throw new Exception("The PGP message is in an unexpected state.");
    }
}