Why is my Java socket receiving an array size larger than what was sent?

Question

I am creating a client/server program to perform encryption with 256 bit AES. I am deriving my key from ECDH. I send the size of the byte arrays I am using to represent my various keys and strings. The issue I am having is that when I try to send the size of my encrypted string from my client to my server, my server says that I sent a much larger size than I actually did. Sending sizes worked for all other byte arrays that needed to be sent. The size of the encrypted string sent from the client is 16 bytes. The server receives an integer size of 276032497 bytes. I have checked that I am in fact sending 16 bytes from the client end.

Any idea of what the issue could be?

Server code:

//generate public key for server
        KeyPairGenerator kpg = KeyPairGenerator.getInstance("EC");
        kpg.initialize(256);
        KeyPair kp = kpg.generateKeyPair();
        byte[] ourPk = kp.getPublic().getEncoded();
        String format = kp.getPublic().getFormat();
        int ourPkLength = ourPk.length;
        
        
        int arrSize;
        
        //send client our pk 
        out.writeInt(ourPkLength);
        out.write(ourPk);
        System.out.println("sent PK!");
        
        //receive pk from client
        arrSize = fromClient.readInt();
        byte[] otherPk = new byte[arrSize];
        fromClient.read(otherPk);
        System.out.println("recived client PK!");
        
        KeyFactory kf = KeyFactory.getInstance("EC");
        X509EncodedKeySpec pkSpec = new X509EncodedKeySpec(otherPk);
        PublicKey otherPublicKey = kf.generatePublic(pkSpec);
    
        //Perform key agreement 
        KeyAgreement ka = KeyAgreement.getInstance("ECDH");
        ka.init(kp.getPrivate());
        ka.doPhase(otherPublicKey, true);
        
        // Send shared secret
        byte[] sharedSecret = ka.generateSecret();
        
        // Derive a key from the shared secret and both public keys
        MessageDigest hash = MessageDigest.getInstance("SHA-256");
        hash.update(sharedSecret);
        // Simple deterministic ordering
        List<ByteBuffer> keys = Arrays.asList(ByteBuffer.wrap(ourPk), ByteBuffer.wrap(otherPk));
        Collections.sort(keys);
        hash.update(keys.get(0));
        hash.update(keys.get(1));

        byte[] derivedKey = hash.digest();
        
        System.out.println("derived key: " + derivedKey + "  length: " + derivedKey.length);
        
        //Convert byte [] to secret key
        //Define cipher
        SecretKeySpec symmetricKey = new SecretKeySpec(derivedKey, 0, 32, "AES");
        Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
        cipher.init(Cipher.DECRYPT_MODE, symmetricKey, new IvParameterSpec(new byte[16]));
        
        //receive encrypted message from client and try to decrypt. 
        arrSize = fromClient.readInt();
        System.out.println("array size sent: " + arrSize);
        byte[] decryptArr = new byte[arrSize];
        fromClient.read(decryptArr);
        System.out.println("Recieved encrypted string: " + decryptArr + "    length:   " + decryptArr.length);
        String decryptStr = Base64.getEncoder().encodeToString(cipher.doFinal(decryptArr));
        
        System.out.println("Decrypted String: " + decryptStr); 

Client code:

 //generate public key for client
        KeyPairGenerator kpg = KeyPairGenerator.getInstance("EC");
        kpg.initialize(256);
        KeyPair kp = kpg.generateKeyPair();
        byte[] ourPk = kp.getPublic().getEncoded();
        //String format = kp.getPublic().getFormat();
        int ourPkLength = ourPk.length;
        
    int arrSize;
    
    //Receive generated public key from the Server
    arrSize = fromServ.readInt();
    byte[] otherPk = new byte[arrSize];
    fromServ.read(otherPk);
    System.out.println("recived server PK!");
    
    //Send the server our public key 
    out.writeInt(ourPkLength);
    out.write(ourPk);
    System.out.println("sent PK!");
    

    
    KeyFactory kf = KeyFactory.getInstance("EC");
    X509EncodedKeySpec pkSpec = new X509EncodedKeySpec(otherPk);
    PublicKey otherPublicKey = kf.generatePublic(pkSpec);
    
    //Perform key agreement 
    KeyAgreement ka = KeyAgreement.getInstance("ECDH");
    ka.init(kp.getPrivate());
    ka.doPhase(otherPublicKey, true);
    
    // Generate a shared secret
    byte[] sharedSecret = ka.generateSecret();
    
    // Derive a key from the shared secret and both public keys
    MessageDigest hash = MessageDigest.getInstance("SHA-256");
    hash.update(sharedSecret);
    // Simple deterministic ordering
    List<ByteBuffer> keys = Arrays.asList(ByteBuffer.wrap(ourPk), ByteBuffer.wrap(otherPk));
    Collections.sort(keys);
    hash.update(keys.get(0));
    hash.update(keys.get(1));

    byte[] derivedKey = hash.digest();   
    System.out.println("derived key: " + derivedKey + "  length: " + derivedKey.length);
    
    //Convert the derivedkey from a byte array to a Secret key Spec of type AES
    SecretKeySpec secretKey = new SecretKeySpec(derivedKey, 0, 32, "AES");
    Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
    cipher.init(Cipher.ENCRYPT_MODE, secretKey, new IvParameterSpec(new byte[16]));
    String plainText = "Testing!";
    byte[] cipherText = cipher.doFinal(plainText.getBytes());
    System.out.println("Encrypted str: " + cipherText + "    length: "+ cipherText.length);
   
    
    //Send encrypted string to Server
    int len = cipherText.length;
    System.out.println("length: " + len);
    out.write(len);
    out.write(cipherText);
    System.out.println("Sent encrypted string!");
    

Answer 1

In your client, you wrote: out.write(len). Your code snippet does not explain what out is, but I'm guessing that .write(someIntValue) is a pass-through from java.io.OutputStream which has that method (.write(int)). The thing is, this writes a single byte, lopping off all bits in your int except the bottom 8.

The matching call in your server code is:

arrSize = fromClient.readInt();

this is not a call that InputStream has (I'm guessing you have some class that extends InputStream and adds these), presumably, what readInt does, is read 4 bytes, and reconstitutes them into a single java int by assuming Big Endian ordering.

So, you send 1 byte from the client and then the string, but the server will read 4 bytes for the length: That 1 byte (the actual length sent by client), plus the first 3 of your string, and tries to interpret that as a length, causing wildly different numbers.

The fact that 276032497 is a number, that, if put in bytes via big endian ordering, starts with a byte with value 16, which is exactly the length you sent, is strongly suggestive that this is your problem.

The fix seems rather trivial; turn out.write(len) into out.writeInt(len). And if you're going to be doing byte-level protocol like this, you need a better testing and debugging plan than ¯\_(ツ)_/¯ I guess I'll ask StackOverflow. This is probably why most folks use different solutions that handrolling raw byte protocols (so, investigate ProtoBuf and friends). At the very least, make the actual pipe a pluggable concept, so that you can plug in a dummy pipe that doesn't encrypt anything, so you can eyeball the bytes flying over the wire to find issues like this; it is unlikely this will be the last time you mismatch your server and client's code.