Java X25519 shared secret is not correct when using testing vectors from RFC7748

Question

When using RFC7748 test vectors for elliptic curve diffie hellman in java, I cannot get expected shared secret key. I am able to do so in other languages. I am using openjdk 11 with default Sun security provider. I found official tests which use these test vectors. But I cannot get expected result even if I copy-paste and run them. For instance, here is test that uses these same vectors which will fail if I copy-paste and run locally. It uses some utility functions which are from here, which I also copied. I know I must be doing something wrong but I cannot figure out what exactly. Here is my code:

public class main {
    public static BigInteger hexStringToBigInteger(boolean clearHighBit, String str) {
        BigInteger result = BigInteger.ZERO;
        for (int i = 0; i < str.length() / 2; i++) {
            int curVal = Character.digit(str.charAt(2 * i), 16);
            curVal <<= 4;
            curVal += Character.digit(str.charAt(2 * i + 1), 16);
            if (clearHighBit && i == str.length() / 2 - 1) {
                curVal &= 0x7F;
                result = result.add(BigInteger.valueOf(curVal).shiftLeft(8 * i));
            }
        }
        return result;
    }

    public static byte[] hexStringToByteArray(String str) {
        byte[] result = new byte[str.length() / 2];
        for (int i = 0; i < result.length; i++) {
            result[i] = (byte) Character.digit(str.charAt(2 * i), 16);
            result[i] <<= 4;
            result[i] += Character.digit(str.charAt(2 * i + 1), 16);
        }
        return result;
    }

    public static String byteArrayToHexString(byte[] arr) {
        StringBuilder result = new StringBuilder();
        for (byte curVal : arr) {
            result.append(Character.forDigit(curVal >> 4 & 0xF, 16));
            result.append(Character.forDigit(curVal & 0xF, 16));
        }
        return result.toString();
    }

    private static void runDiffieHellmanTest(String curveName, String a_pri,
                                             String b_pub, String result) throws Exception {

        NamedParameterSpec paramSpec = new NamedParameterSpec(curveName);
        KeyFactory kf = KeyFactory.getInstance("XDH");
        KeySpec privateSpec = new XECPrivateKeySpec(paramSpec, hexStringToByteArray(a_pri));
        PrivateKey privateKey = kf.generatePrivate(privateSpec);
        boolean clearHighBit = curveName.equals("X25519");
        KeySpec publicSpec = new XECPublicKeySpec(paramSpec, hexStringToBigInteger(clearHighBit, b_pub));
        PublicKey publicKey = kf.generatePublic(publicSpec);

        byte[] encodedPrivateKey = privateKey.getEncoded();
        System.out.println("Encoded private: " + byteArrayToHexString(encodedPrivateKey));
        byte[] encodedPublicKey = publicKey.getEncoded();
        System.out.println("Encoded public: " + byteArrayToHexString(encodedPublicKey));

        KeyAgreement ka = KeyAgreement.getInstance("XDH");
        ka.init(privateKey);
        ka.doPhase(publicKey, true);

        byte[] sharedSecret = ka.generateSecret();
        byte[] expectedResult = hexStringToByteArray(result);
        if (!Arrays.equals(sharedSecret, expectedResult)) {
            throw new RuntimeException("fail: expected=" + result + ", actual="
                    + byteArrayToHexString(sharedSecret));
        }
    }

    public static void main(String[] args) throws Exception {
        runDiffieHellmanTest(
                "X25519",
                "77076d0a7318a57d3c16c17251b26645df4c2f87ebc0992ab177fba51db92c2a",
                "de9edb7d7b7dc1b4d35b61c2ece435373f8343c85b78674dadfc7e146f882b4f",
                "4a5d9d5ba4ce2de1728e3bf480350f25e07e21c947d19e3376f09b3c1e161742");
    }
}

Answer 1

You have incorrectly swapped two lines in hexStringToBigInteger:

            if (clearHighBit && i == str.length() / 2 - 1) {
                curVal &= 0x7F;
                result = result.add(BigInteger.valueOf(curVal).shiftLeft(8 * i));
            }

should instead be:

            if (clearHighBit && i == str.length() / 2 - 1) {
                curVal &= 0x7F;
            }
            result = result.add(BigInteger.valueOf(curVal).shiftLeft(8 * i));

Answer 2

I could not establish shared secret between Go and Java applications, so I tried to debug what was the cause of that which led me into reading RFC7748 and digging through source code for X25519 key exchange in Java. So for people who want to perform X25519 key exchange between Java and some other non-Java application - here is main takeaway. Java already expects input public key to be BigInteger instead of byte array. Some other languages may return public key as a byte array in big-endian format. Due to RFC7748 specification, X coordinate of a point on elliptic curve (which is your public key byte array) must be in little-endian format. So you only have to reverse input public key byte array to make it little-endian, before feeding in to BigInteger.