ECDSA signature generated with mbedtls not verifiable in JOSE (while code worked with RSA key)

Question

I have a small application running on an ESP32 dev board (I use the Arduino IDE together with the shipped mbedtls) that issues and verifies JWT tokens. I have at first used RSA signatures successfully, but now wanted to go for shorter signatures and thus attempted to use ECDSA. The application itself can issue tokens and verify them as well, but if I attempt to verify the tokens outside of my application - for instance with JOSE or the Debugger- I get verification failures and I cant quite wrap my head around why this is happening.

This is an example token (this token does practically not contain information):

eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE2NzI0MTMxNjgsImV4cCI6MTY3MjQxNjc2OH0.MEUCIAjwEDXI424qjrAkSzZ_ydcVLOSAvfQ8YVddYvzDzMvQAiEAkVy4d-hZ01KpcMNKhPHk8E_SDYiB4JKwhm-Kc-Z81rI

This is the corresponding public key (this key is not used anywhere besides the purpose of presenting the issue here):

-----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEUGnNIOhPhZZSOg4A4BqAFtGO13W4BGDQpQ0ieTvLU9/CXrY7W77o7pNx7tvugeIoYJxS0NjmxvT4TMpo4Z8P7A== -----END PUBLIC KEY-----

As far as I understand, JWT tokens can be issued and verified using ECDSA. The so called "ES256" method is supposed to use prime256v1 in combination with SHA256, so I generated my key material with the following commands:

openssl ecparam -name prime256v1 -genkey -noout -out ecc-private.pem
openssl ec -in ecc-private.pem -pubout -out ecc-public.pem

For the signing part, the private key is loaded as follows, where ecc_priv is a String containing the PEM representation of the key:

//get the key
byte *keybuffer = (byte*)malloc((ecc_priv.length()+1)*sizeof(byte));
ecc_priv.getBytes(keybuffer, ecc_priv.length() + 1);
mbedtls_pk_context pk_context;
mbedtls_pk_init(&pk_context);
int rc = mbedtls_pk_parse_key(&pk_context, keybuffer, ecc_priv.length() + 1, NULL, 0);
if (rc != 0){
 printf("Failed to mbedtls_pk_parse_key: %d (-0x%x): %s\n", rc, -rc, mbedtlsError(rc));
 return -1;
}
free(keybuffer);

Since this worked for me with RSA keys, I just replaced the keys and kept all other code to sign the actual message. As far as I understand, this should be possible with mbedtls_pk methods:

//mbedtls context
mbedtls_entropy_context entropy;
mbedtls_ctr_drbg_context ctr_drbg;
mbedtls_ctr_drbg_init(&ctr_drbg);
mbedtls_entropy_init(&entropy);

const char* pers="some entropy";
                
mbedtls_ctr_drbg_seed(
  &ctr_drbg,
  mbedtls_entropy_func,
  &entropy,
  (const unsigned char*)pers,
  strlen(pers));
//get the header and payload bytes    
byte *headerAndPayloadbytes = (byte*)malloc((headerAndPayload.length()+1)*sizeof(byte));
headerAndPayload.getBytes(headerAndPayloadbytes, headerAndPayload.length() + 1);
//prepare digest
uint8_t digest[32];
rc = mbedtls_md(mbedtls_md_info_from_type(MBEDTLS_MD_SHA256), headerAndPayloadbytes, headerAndPayload.length(), digest);
if (rc != 0) {
  printf("Failed to mbedtls_md: %d (-0x%x): %s\n", rc, -rc, mbedtlsError(rc));
  return -1;        
}
free(headerAndPayloadbytes);
//prepare output
byte *oBuf = (byte*)malloc(5000*sizeof(byte));
size_t retSize;
//sign digest
rc = mbedtls_pk_sign(&pk_context, MBEDTLS_MD_SHA256, digest, sizeof(digest), oBuf, &retSize, mbedtls_ctr_drbg_random, &ctr_drbg);
if (rc != 0) {
  printf("Failed to mbedtls_pk_sign: %d (-0x%x): %s\n", rc, -rc, mbedtlsError(rc));
  return -1;        
}
//encode signature to base64
unsigned int osize = encode_base64_length(retSize);
byte *output = (byte*)malloc((osize+1)*sizeof(byte));
encode_base64(oBuf, retSize, output);
String sig = String((char*)output);
free(output);
//base64 URL specific
sig.replace('+','-');
sig.replace('/','_');
sig.replace("=","");
String completejwt = headerAndPayload + "." + sig;
//free resources
mbedtls_ctr_drbg_free( &ctr_drbg );
mbedtls_entropy_free( &entropy );
mbedtls_pk_free(&pk_context);
free(oBuf);

My expectation was that I can simply replace the RSA keys with the ECDSA (prime256v1) keys and keep everything else as is, but the resulting token are not verifiable outside of my application. Again I want to emphasize that inside my application I can definitely verify the token and that the code worked perfectly fine with RSA keys, even outside my application. There must be something Im missing here, Im sure about that. Any help or directions to research are highly appreciated.

EDIT: Here is a minimal compilable example (Arduino sketch)

Answer 1

Your ECDSA signature is a DER-encoded ASN.1 structure, rather than a simple r || s concatenation as proposed by IEEE-P1363 which is what the JOSE specification mandates.

Answer 2

Try to sign in P1363 format using mbedtls_ecdsa_sign_det() method

    auto ecdsa = mbedtls_pk_ec(pkContext);

    if (ecdsa == NULL)
    {
        return String("INVALID-ECDSA-CONTEXT");
    }

    unsigned char signature[64];
    size_t signatureLength;
    mbedtls_mpi r, s;

    mbedtls_mpi_init(&r);
    mbedtls_mpi_init(&s);

    
    int ret = mbedtls_ecdsa_sign_det(&ecdsa->grp, &r, &s, &ecdsa->d, sha256_b64h_b64p, 32, MBEDTLS_MD_SHA256);
    if (ret != 0)
    {
        return String("CAN-NOT-SIGN");
    }

    // Write the signature in P1363 format
    mbedtls_mpi_write_binary(&r, signature, mbedtls_mpi_size(&r));
    mbedtls_mpi_write_binary(&s, signature + mbedtls_mpi_size(&r), mbedtls_mpi_size(&s));
    signatureLength = mbedtls_mpi_size(&r) + mbedtls_mpi_size(&s);

    /* mbedtls_ecdsa_write_signature writes signature out as DER format */
    // int success = mbedtls_ecdsa_write_signature(ecdsa, MBEDTLS_MD_SHA256, sha256_b64h_b64p, 32, buf, &signatureLength, NULL, NULL);

    /* Print signature as hex */
     Serial.print("signature (hex) (" + String(signatureLength) + ") : [");
     for (size_t i = 0; i < signatureLength; i++)
     {
         Serial.printf("%02x", signature[i]);
     }
     Serial.println("]");