SSL_HANDSHAKE Error Domino TLS Outgoing

4.9k views Asked by At

We have 5 customers running the same WebService from Domino This weekend we updated the customers servers with Domino 9.01. FP2 and the Poodle fixpack to be able to run TLS 1.0 incomming and outgoing.

4 Customers works perfect 1 Customer gets SSL errors for the outgoing Webservice (same errors as before we updated the servers), the incomming is working for TLS so we guess the updates for Poodle have worked as intended.

After setting som DEBUG_SSL parameters for one working and the failing server we got this logs

The failing row is

S_Read> nti_done return 0 bytes rc = 9

instead of intended

S_Read> nti_done return 5 bytes rc = 0

SSL_RCV> 00000000: 16 03 01 00 2E

I have searched google and nothing is there to understand what is missing My guess is there is some problem with negotiating the cipher, but why and what to do for solving this matter. I know there is some smart people out there ;-)

Log from failing server handshake

  int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
  SSL_Handshake> Enter
  SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)
  SSLAdvanceHandshake Enter> Processed : 0 State: 4 (HandshakeClientIdle)
  SSLAdvanceHandshake Enter> Processed : SSL_hello_request
  SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeClientHello
  SSLEncodeClientHello> We offered SSL/TLS version TLS1.0 (0x0301)
  SSLAdvanceHandshake Exit> State : 5 (HandshakeServerHello)
  S_Write> Enter len = 58
  SSL_Xmt> 00000000: 16 03 01 00 35 01 00 00 31 03 01 54 A5 85 B7 4D   '....5...1..T%.7M'
  SSL_Xmt> 00000010: 15 80 11 80 C7 47 4D 1D 1D B1 89 5F F6 94 18 73   '....GGM..1._v..s'
  SSL_Xmt> 00000020: C6 D3 7D 6A 15 92 A9 57 48 19 32 00 00 0A 00 2F   'FS}j..)WH.2..../'
  SSL_Xmt> 00000030: 00 35 00 05 00 0A 00 04 01 00                     '.5........'
  S_Write> Switching Endpoint to sync
  S_Write> Posting a nti_snd for 58 bytes
  SSL_EncryptData> SSL not init exit
  S_Write> Switching Endpoint to async
  SSL_EncryptDataCleanup> SSL not init exit
  S_Write> nti_done return 58 bytes rc = 0
  S_Write> Exit, wrote 58 bytes
  S_Read> Enter len = 5
  S_Read> Switching Endpoint to sync
  S_Read> Posting a nti_rcv for 5 bytes
  SSL_RcvSetup> SSL not init exit
  S_Read> Switching Endpoint to async
  S_Read> nti_done return 0 bytes rc = 9
  S_Read> nti_done return 0 bytes rc = 9 Event = 0x100
  SSLSendAlert> Sending an alert of 0x0 (close_notify) level 0x2 (fatal)
  SSL_Handshake> Changing SSL status from -6989 to -5000 to flush write queue
  SSL_Handshake> After handshake state= 2 Status= -5000
  SSL_Handshake> Exit Status = -5000
  int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]

...

Log from working server handshake

  int_MapSSLError> Mapping SSL error 0 to 0 [SSLNoErr]
  SSL_Handshake> Enter
  SSL_Handshake> Current Cipher 0x0000 (Unknown Cipher)
  SSLAdvanceHandshake Enter> Processed : 0 State: 4 (HandshakeClientIdle)
  SSLAdvanceHandshake Enter> Processed : SSL_hello_request
  SSLAdvanceHandshake calling SSLPrepareAndQueueMessage> SSLEncodeClientHello
  SSLEncodeClientHello> We offered SSL/TLS version TLS1.0 (0x0301)
  SSLAdvanceHandshake Exit> State : 5 (HandshakeServerHello)
  S_Write> Enter len = 58
  SSL_Xmt> 00000000: 16 03 01 00 35 01 00 00 31 03 01 54 A5 89 B3 A0   '....5...1..T%.3 '
  SSL_Xmt> 00000010: 2B 75 D1 E9 D4 81 87 C3 5D 91 45 84 6A E2 47 9D   '+uQiT..C].E.jbG.'
  SSL_Xmt> 00000020: 76 BE 14 A8 A6 10 1C 06 FB 7D 8B 00 00 0A 00 2F   'v>.(&...{}...../'
  SSL_Xmt> 00000030: 00 35 00 05 00 0A 00 04 01 00                     '.5........'
  S_Write> Switching Endpoint to sync
  S_Write> Posting a nti_snd for 58 bytes
  SSL_EncryptData> SSL not init exit
  S_Write> Switching Endpoint to async
  SSL_EncryptDataCleanup> SSL not init exit
  S_Write> nti_done return 58 bytes rc = 0
  S_Write> Exit, wrote 58 bytes
  S_Read> Enter len = 5
  S_Read> Switching Endpoint to sync
  S_Read> Posting a nti_rcv for 5 bytes
  SSL_RcvSetup> SSL not init exit
  S_Read> Switching Endpoint to async
  S_Read> nti_done return 5 bytes rc = 0
  SSL_RCV> 00000000: 16 03 01 00 2E                                    '.....'
  S_Read> Exit, read 5 bytes
  S_Read> Enter len = 46
  S_Read> Switching Endpoint to sync
  S_Read> Posting a nti_rcv for 46 bytes
  SSL_RcvSetup> SSL not init exit
  S_Read> Switching Endpoint to async
  S_Read> nti_done return 46 bytes rc = 0
  SSL_RCV> 00000000: 02 00 00 2A 03 01 54 7C 9D 24 4C B4 AD 62 4E 35   '...*..T|.$L4-bN5'
  SSL_RCV> 00000010: 4C C3 B4 AB 34 6D 7D CB 8F 6B CC 80 00 FE 4C 4A   'LC4+4m}K.kL..~LJ'
  SSL_RCV> 00000020: 77 87 CD 2E DF 98 04 10 13 29 0B 00 2F 00         'w.M._....)../.'
  S_Read> Exit, read 46 bytes
  SSLProcessProtocolMessage> Record Content: 22
  SSLProcessHandshakeMessage Enter> Message: 2 State: 5 (HandshakeServerHello) Key Exchange: 0 Cipher: 0x0000 (Unknown Cipher)
  SSLProcessHandshakeMessage Enter> Message: SSL_server_hello
  SSLProcessServerHello> Server chose SSL/TLS version TLS1.0 (0x0301)
  SSLProcessHandshakeMessage Exit> Message: 2 State: 5 (HandshakeServerHello) Key Exchange: 1 Cipher: 0x002F (RSA_WITH_AES_128_CBC_SHA)
  SSLAdvanceHandshake Enter> Processed : 2 State: 5 (HandshakeServerHello)
  SSLAdvanceHandshake Enter> Processed : SSL_server_hello
  SSLAdvanceHandshake Exit> State : 8 (HandshakeCertificate)
  SSL_Handshake> After handshake state= 8 Status= -5000
  SSL_Handshake> Exit Status = -5000
  int_MapSSLError> Mapping SSL error -5000 to 4176 [SSLHandshakeNoDone]
  SSL_Handshake> Enter
  SSL_Handshake> Current Cipher 0x002F (RSA_WITH_AES_128_CBC_SHA)
  S_Read> Enter len = 5
  S_Read> Switching Endpoint to sync
  S_Read> Posting a nti_rcv for 5 bytes
  SSL_RcvSetup> SSL not init exit
  S_Read> Switching Endpoint to async
  S_Read> nti_done return 5 bytes rc = 0
  SSL_RCV> 00000000: 16 03 01 0E 9D                                    '.....'
  S_Read> Exit, read 5 bytes
  S_Read> Enter len = 3741

....

/Stefan

PS: Here is the Java errors that come sfter the hand shake error

Error connecting to 'xxxxx' on port '443', SSL IO error. Remote session no longer responding.
at lotus.domino.axis.InternalFault.makeFault(Unknown Source)
at lotus.domino.axis.transport.http.HTTPSender.invoke(Unknown Source)
at lotus.domino.axis.strategies.InvocationStrategy.visit(Unknown Source)
at lotus.domino.axis.SimpleChain.doVisiting(Unknown Source)
at lotus.domino.axis.SimpleChain.invoke(Unknown Source)
at lotus.domino.axis.client.AxisClient.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invokeEngine(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.axis.client.Call.invoke(Unknown Source)
at lotus.domino.websvc.client.Call.invoke(Unknown Source)
0

There are 0 answers