I need to create and install a self-signed certificate on the server (an XML hardware appliance) to do SSL authentication of a Java client/application which, through its interface configuration, can set keystores, i.e. .jks. I only need this setup for testing purposes and not production, for obvious reasons. Here's how I am creating my keystore (PKCS12 type):
1) Generate a private key (pair)
$> openssl genrsa -des3 -out private.key 1024
2) Generate a certificate request (is this needed?) -- I only set value to CN
$> openssl req -new -key private.key -out request.csr
3) Create a self-signed certificate
$> openssl x509 -req -days 365 -in request.csr -signkey private.key -out cert.crt
4) Create PKCS12 keystore
$> openssl pkcs12 -export -in cert.crt -inkey private.key -out keystore.p12
5) Create client JKS
$> keytool -importkeystore -srckeystore keystore.p12 -srcstoretype pkcs12 -srcalias myalias -destkeystore client.jks -deststoretype jks -deststorepass <same pass as private key> -destalias myalias
When I install keystore.p12 on the server/appliance, client.jks on the client Java application, and make a request I get the following error:
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
Am I fundamentally missing something about certificates and SSL authentication?
Not an answer yet, but getting too complicated for comments so I'll give a start and edit later.
Should (SSL/TLS) server(s) and client(s) share a key (and cert)? Okay for development and maybe test, varies for production. As a general rule every independent system that needs to be authenticated should have its own privatekey and a certificate for that key (or sometimes but rarely multiple certs). If both (or more generally all) the systems involved and the communication between them will be under control of the same administrator, it doesn't really hurt to relax this rule and (re)use one key+cert. Since the key+cert(s) used should always be (re)configurable, this is a decision you can change later, even change several times if that turns out to be desirable or necessary.
Should the cert(s) be self-signed? Usually good for development and test, varies for production. Again if all relevant systems are under control of one admin, or at least a limited group of people who know each other (such as divisions or offices of one organization) there's no real security need for a CA to verify identities and determine trust. One consideration is that the server cert must encode the domain-name(s) (or IP address(es) if you use that option) in the cert, so changing them, which is often needed in development and test, means going back to the CA for a new cert, at least a minor nuisance and sometimes a delay.
At the moment you have one key and selfsigned cert you are trying to share for both client and server. While you may well want to change these later, I would recommend focussing on getting something working first. Changing several interrelated things at the same time tends to cause confusion.
"Extract" a cert? For the current setup, with one (shared) key+cert (selfsigned), you already have the key+cert in a JKS, the same key+cert in a P12, and the same cert in a file (cert.crt) so you don't need to extract anything. If you (later) generate for the Java (client) side a new key and selfsigned cert using keytool, yes you would need to extract that cert. If you obtain a CA-signed cert for either the existing key or a new key, you might need to extract the root/anchor, but you might already have it, see below.
What cert(s) is needed to trust? The client always (with minor exceptions not relevant here) needs in its truststore the anchor for the server's cert, and when client authentication (also called mutual or two-way) is used as in your case the server needs in its truststore the anchor for the client's cert. In your current case the client and server use the same cert, so both sides need the same anchor.
For a self-signed cert the anchor must be the cert itself -- which must be changed, usually manually, any time the partner cert changes.
For a CA-issued cert the anchor can be and (always?) should be the root cert for that CA; the CA root is usually long-lived (like 20 years and more) and does not need to change when a partner/entity cert does.
If you use a well-known CA like Verisign or GoDaddy, as opposed to a CA run by yourself or maybe by "Joe's Dicount Liquor Store and Certificate Authority" in your local red-light district, in some systems the roots for those well-known CAs may be already installed in which case you don't need to do anything.
How did the client work to trust the server? Your Java client apparently used the client.jks file as both keystore and truststore. This can easily work because Java uses one file format that can contain both kinds of data, and further treats the cert in an own-key entry as also a trusted cert (anchor).
What about the server truststore? On the other hand, software that uses pkcs12 format for a keystore, or at least to import a keystore entry, sometimes (I'd say often) does not use that same format for its truststore, and certainly not the same file(s). The fact that it did reject client use of the same cert it already has as its own probably means it doesn't treat the pkcs12 you gave it as truststore data, although in principle it could be disliking something else such as the lack of an ExtendedKeyUsage extension or the unavailability of revocation/status data.
You don't identify or describe the device which is your server, so I can only guess at the myriad of ways that SSL/TLS servers, especially embedded ones, can work. But somewhere in your device there is probably a way to add certs to its truststore. Also it should have some kind of error log somewhere that may contain additional information about whether it just needs the anchor (here the selfsigned cert) in the truststore, or something else -- assuming you can see that log without SSL/TLS authentication!
If you can give us (a link to) doc on your server/device, or tell us in sufficient detail what it shows, offers, and/or accepts, I can probably be more specific.