Exception when using SSLEngine

Question

I am writing a custom server for an android phone to handle https requests. It starts by listening on a specific port and once a connection is made, it performs handshake with the client. Here is the snipet of the code:

ServerSocket sock = new ServerSocket(8080);
Socketclient client = sock.accept();
doHandshake(client);

...

void doHandshake(Socket socket) throws Exception {
        try {

             SSLContext context = SSLContext.getDefault();
             SSLEngine engine = context.createSSLEngine();
             engine.setUseClientMode(false);

            SSLSession session = engine.getSession();
            // Create byte buffers to use for holding application data
            ByteBuffer myAppData = ByteBuffer.allocate(session.getApplicationBufferSize());
            ByteBuffer peerAppData = ByteBuffer.allocate(session.getApplicationBufferSize());

            ByteBuffer myNetData = ByteBuffer.allocate(session.getPacketBufferSize());
            byte[] peerNetData = new byte[session.getPacketBufferSize()];

            // Begin handshake
            engine.beginHandshake();
            SSLEngineResult.HandshakeStatus hs = engine.getHandshakeStatus();
            InputStream inputStream = socket.getInputStream();
            OutputStream outputStream = socket.getOutputStream();

            int bytesRead;
            // Process handshaking message
            while (hs != SSLEngineResult.HandshakeStatus.FINISHED &&
                    hs != SSLEngineResult.HandshakeStatus.NOT_HANDSHAKING) {

                switch (hs) {

                    case NEED_UNWRAP:
                        // Receive handshaking data from peer
                        bytesRead = inputStream.read(peerNetData);
                        if (bytesRead < 0) {
                            // The channel has reached end-of-stream
                        }

                        ByteBuffer peerData = ByteBuffer.wrap(peerNetData, 0, bytesRead);

                        SSLEngineResult res = engine.unwrap(peerData, peerAppData);

When I pointed the browser to this seerver (url = https://localhost:8080) I got an exception on "engine.unwrap(peerData, peerAppData)"

javax.net.ssl.SSLProtocolException: SSL handshake terminated: ssl=0xb8347db8: Failure in SSL library, usually a protocol error error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher (external/openssl/ssl/s3_srvr.c:1394 0xace00e61:0x00000000)

What am I missing?

Answer 1

You should throw away this code. Get rid of the SSLEngine and just use SSLSocket if you're going to use streams. The SSLEngine is for use in non-blocking mode with SocketChannels, and it's hard enough to get that right without adding streams into the mix.

Answer 2

no shared cipher

There is a problem with setting up you SSL engine properly. One reason might be that you don't set up any certificates and thus only anonymous cipher suites can be used. But the browser does not support these insecure cipher suites and this there is no shared cipher suite between browser and server.