Java 11 Curve25519 Implementation doesn't behave as Signal's libary

Question

In Java 11 a curve25519 built-in implementation was introduced. As I had no idea of this, and only discovered it recently, I was using a library from Signal. This was my code before I switched to Java 11's implementation:

private final Curve25519 CURVE_25519 = Curve25519.getInstance(Curve25519.JAVA);

public Curve25519KeyPair calculateRandomKeyPair() {
    return CURVE_25519.generateKeyPair();
}

public byte[] calculateSharedSecret(byte[] publicKey, byte[] privateKey) {
    return CURVE_25519.calculateAgreement(publicKey, privateKey);
}

And this is my code now:

private final String XDH = "XDH";
private final String CURVE = "X25519";

@SneakyThrows
public KeyPair calculateRandomKeyPair() {
    return KeyPairGenerator.getInstance(CURVE).generateKeyPair();
}

@SneakyThrows
public byte[] calculateSharedSecret(byte[] publicKeyBytes, XECPrivateKey privateKey) {
    var paramSpec = new NamedParameterSpec(CURVE);
    var keyFactory = KeyFactory.getInstance(XDH);

    var publicKeySpec = new XECPublicKeySpec(paramSpec, new BigInteger(publicKeyBytes));
    var publicKey = keyFactory.generatePublic(publicKeySpec);

    var keyAgreement = KeyAgreement.getInstance(XDH);
    keyAgreement.init(privateKey);
    keyAgreement.doPhase(publicKey, true);
    return keyAgreement.generateSecret();
}

Obviously, the second implementation doesn't work while the first one does. Initially, I thought I was doing something wrong so I read the documentation and checked similar answers, though, as I didn't find anything helpful, I decided to dig further and tried to check if both Signal's library and Java generate the same public key given the private one. To do this I wrote this snippet:

import org.whispersystems.curve25519.Curve25519;
import sun.security.ec.XECOperations;
import sun.security.ec.XECParameters;

import java.security.InvalidAlgorithmParameterException;
import java.security.spec.NamedParameterSpec;
import java.util.Arrays;

private static boolean generateJava11KeyPair() throws InvalidAlgorithmParameterException {
    var signalKeyPair = Curve25519.getInstance(Curve25519.JAVA).generateKeyPair();
    
    var signalPublicKey = signalKeyPair.getPublicKey();
    var params = XECParameters.get(InvalidAlgorithmParameterException::new, NamedParameterSpec.X25519);
    var ops = new XECOperations(params);
    var javaPublicKey = ops.computePublic(signalKeyPair.getPrivateKey().clone()).toByteArray();
    
    return Arrays.equals(signalPublicKey, javaPublicKey);
}

(the code used to calculate the public key following Java's implementation was extracted from sun.security.ec.XDHKeyPairGenerator)

This method prints false, which means that the two implementations actually don't behave in the same way. At this point, I'm wondering if this is a Java bug or if I'm missing something. Thanks in advance.

Answer 1

The encoding defined by Bernstein et al for X25519 (and X448) keys both public and private is unsigned fixed-length little-endian, while the representation returned by BigInteger.toByteArray() and accepted by ctor BigInteger(byte[]) is twos-complement variable-length big-endian. Since 255 bits rounds up to 32 bytes with a spare bit that is always zero (for XDH) the signedness difference can be ignored there, but the others matter.

JCA did make the inteface class XECPrivateKey return, and the corresponding Spec accept, these forms, but for XECPublicKey[Spec] it uses BigInteger. It does use the Bernstein forms consistently for (both) the "X509" and "PKCS8" encodings (respectively) returned by Key.getEncoded() and accepted by a KeyFactory, but those have metadata that XDH-only (or Bernstein-only XDH-and-EdDSA) systems like X3DH don't use.

AFAICS your choices are

• byte-reverse and (zero-)pad when needed the JCA public values in your code, or
• use Key.getEncoded() and parse the algorithm-specific part or conversely build the algorithm-generic structure to pass as X509EncodedKeySpec to KeyFactory.getInstance("Xblah").

The second approach has been asked about in the past for other algorithms: 'traditional' (X9-style) EC -- especially secp256k1 for bitcoin and related coins, which generally use only the raw-X9/SECG data with no metadata -- and RSA where a few systems use the raw-PKCS1 formats (here more commonly for privatekey than publickey); if you want I can find some near-duplicates to illustrate the approach.