Leshan demo server provided a root CA
If you want to connect a client using DTLS with certificate(x509) mode, your client need to trust this certificate to accept DTLS connection with this server.
What i want to do is make a new x509 certificate signed by the Root CA provided by the Leshan demo server. This root CA is in der format and i don't know how to use it with openSSL.
I tried to use openssl ca but it said unable to load CA private key.
To sign a certificate, you need the private key of the signer CA. The public key (maybe from the signing CA certificate) is then used to verify that sign.
In my opinion, it's a very, very common misinterpretation, that x509 improves the security for IoT. Especially, if it's not that clear, how it works, it usually cause more trouble then it helps. CoAP/DTLS offers also other ways, e.g. Raw Public Key (RFC 7250) or PSK-ECDHE. Anyway, if you want/must go with x509, let me recommend, that you get more common with it ahead.
Just to mention: If you want in the end to sign your device certificate, you may sign that by a different CA, it's not required, that this is the CA used to sign the server's certificate. Maybe the leshan sandbox requires that, but that's more a question to the operator of that sandbox (ask the leshan project about that).
For Eclipse/Californium, it's easier to create the device certificate with the java keytool and export it afterwards. Some help create-keystores.sh, if you read that shell script carefully.