Throwing exception on NIO SSL Handshake process using SSLEngine

Question

I am getting following exception on NIO SSL handshake. During handshake process,

On client side,

a) NEED_WRAP

b) NEED_UNWRAP

c) NEED_TASK

d) NEED_UNWRAP - getting the following exception on calling unwrap.

javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 1
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1371)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:790)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at com.ipay.ssl.SSlClientNio.doHandshake(SSlClientNio.java:65)
    at com.ipay.ssl.SSlClientNio.main(SSlClientNio.java:220)
javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 1
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:133)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:808)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:806)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1299)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Please tell me if handshake sequence i am following is correct. I am completely stuck on handshaking part.

//Editied

Following is the handshake code.

void doHandshake(SelectionKey key, SSLEngine engine,
            ByteBuffer myNetData, ByteBuffer peerNetData) throws Exception {

        SocketChannel socketChannel = (SocketChannel)key.channel();
        // Create byte buffers to use for holding application data
        int appBufferSize = engine.getSession().getApplicationBufferSize();
        ByteBuffer myAppData = ByteBuffer.allocate(appBufferSize);

        // Begin handshake
        engine.beginHandshake();
        SSLEngineResult.HandshakeStatus hs = engine.getHandshakeStatus();
        System.out.println("handshake"+hs);

        // Process handshaking message
        while (hs != SSLEngineResult.HandshakeStatus.FINISHED &&
            hs != SSLEngineResult.HandshakeStatus.NOT_HANDSHAKING) {
             switch (hs) {
                case NEED_UNWRAP:
                System.out.println("Reached NEED UNWRAP");
                // Receive handshaking data from peer
                if (socketChannel.read(myNetData) < 0) {
                    // Handle closed channel
                    System.out.println("not able toRead data from channel to buffer at client");
                }
                myNetData.flip();
                // Process incoming handshaking data
                if(myNetData.limit() == 0)
                {
                    myNetData.clear();
                    break;
                }
                myAppData.clear();
                System.out.println("checking limit"+myNetData.limit());
                SSLEngineResult res = engine.unwrap(myNetData, myAppData);
                myNetData.compact();
                // Getting handshake status
                hs = res.getHandshakeStatus();
                System.out.println("Debugging in NEED_UNWRAP-->"+hs);
                // Check status
                switch (res.getStatus()) {
                case OK :
                    // Handle OK status
                    System.out.println("OK");
                    break;
                case BUFFER_OVERFLOW:
                    System.out.println("BUFFER OVERFLOW");
                    break;
                case BUFFER_UNDERFLOW:
                    System.out.println("BUFFER UNDERFLOW");
                    break;
                case CLOSED:
                    System.out.println("CLOSED");
                    break;

                // Handle other status: BUFFER_UNDERFLOW, BUFFER_OVERFLOW, CLOSED
               // ...
                }
                break;
            case NEED_WRAP :
                System.out.println("Reached NEED WRAP");
                // Empty the local network packet buffer.
                myNetData.clear();
                // Generate handshaking data
                res = engine.wrap(myAppData, myNetData);
                // Getting handshake status
                hs = res.getHandshakeStatus();
                System.out.println("Debugging in NEED_WRAP-->"+hs);
                System.out.println(engine.getSession().getApplicationBufferSize());
                System.out.println(myNetData.capacity());
                // Check status
                switch (res.getStatus()) {
                case OK :
                    System.out.println("OK");
                    myNetData.flip();

                    // Send the handshaking data to peer
                    while (myNetData.hasRemaining()) {
                        if (socketChannel.write(myNetData) < 0) {
                            // closing socket channel
                        }
                    }
                    break;
                case BUFFER_OVERFLOW:
                    System.out.println("BUFFER OVERFLOW");
                    break;
                case BUFFER_UNDERFLOW:
                    System.out.println("BUFFER UNDERFLOW");
                    break;
                case CLOSED:
                    System.out.println("CLOSED");
                    break;
                // Handle other status:  BUFFER_OVERFLOW, BUFFER_UNDERFLOW, CLOSED
               // ...
                }
                    break;
            case NEED_TASK :
                System.out.println("NEED TASK");
                System.out.println("Debugging in NEED_TASK-->"+hs);
                Runnable task;

                while((task=engine.getDelegatedTask()) != null)
                {
                System.out.println("Inside while loop");
                ExecutorService executorService = Executors.newFixedThreadPool(1);
                executorService.execute(task);

                // Handle blocking tasks
                }
                // Whether following code is required
                hs=engine.getHandshakeStatus();
                System.out.println("Printing"+engine.getHandshakeStatus());
                break;
            case FINISHED:
                System.out.println("Debugging in FINISHED-->"+hs);
                System.out.println("handshake done");
                break;

            //...
            }
        }

        // Processes after handshaking
        //...
    }

//UPDATED EXCEPTION...

I am getting following exception on server side. I googled it.But getting no idea.

Please help me on this part..

javax.net.ssl.SSLProtocolException: Handshake message sequence violation, state = 1, type=1   
javax.net.ssl.SSLProtocolException: Handshake message sequence violation, state = 1, type = 1        

    at sun.security.ssl.Handshaker.checkThrown(Unknown Source)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(Unknown Source)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source)
    at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
    at javax.net.ssl.SSLEngine.unwrap(Unknown Source)
    at com.ipay.ssl.SSLServerNio.doHandshake(SSLServerNio.java:55)
    at com.ipay.ssl.SSLServerNio.main(SSLServerNio.java:216)
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, state = 1, type = 1
    at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
    at sun.security.ssl.Handshaker.processLoop(Unknown Source)
    at sun.security.ssl.Handshaker$1.run(Unknown Source)
    at sun.security.ssl.Handshaker$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

varghese

Answer 1

You're not supposed to 'follow' a 'sequence' when using the SSLEngine. You are supposed to react to the states and exceptions it provides:

• NEED_WRAP: do a wrap() from the application send buffer to the network send buffer
• NEED_UNWRAP: do an unwrap() from the network receive buffer to the application receive buffer
• BUFFER_OVERFLOW: do a flip()/write()/compact() on the network send buffer, or a flip()/get()/compact() on the application receive buffer
• BUFFER_UNDERFLOW: do a read() on the network receive buffer, or there is nothing in the application send buffer.

EDIT What's this?

if(myNetData.limit() == 0)
{
    myNetData.clear();
}

and this?

case NEED_WRAP:
    myNetData.clear();

You can't just throw away engine data. Get rid of these altogether. You can't do anything to the network send or receive buffers except flip(), compact(), wrap(), and unwrap(). Also it doesn't appear that you have separate net send and receive buffers. You need both. You need four altogether: net send, net receive, application send, and application receive. The network buffers need to be of the sizes advised by the SSLEngine.

Apart from that, you aren't really reacting exactly as I said above. For example, take the NEED_UNWRAP path. You should:

1. unwrap() from the net recv buffer to the application receive buffer.
2. If you then get BUFFER_UNDERFLOW, read() into the net receive buffer and repeat (1).

For NEED_WRAP:

3. wrap() from the application send buffer to the network send buffer.
4. If you then get BUFFER_OVERFLOW, write() from the net send buffer and repeat 3.

When you need to read application data:

5. flip()/get()/compact() from the application receive buffer.
6. If that results in a BufferUnderflowException, unwrap() and repeat, bearing in mind that the unwrap() may cause NEED_WRAP or NEED_WRAP or BUFFER_UNDERFLOW or BUFFER_OVERFLOW.

When you need to write application data:

7. put() into the application send buffer.
8. If that results in a BufferOverflowException, flip()/wrap()/compact(), bearing in mind that the wrap() may cause NEED_WRAP or NEED_WRAP or BUFFER_UNDERFLOW or BUFFER_OVERFLOW.