curl command : remote server is not able to respond after TLS handshake in google compute node

Question

In a google compute node, when I run this command curl -v https://www1.nseindia.com/, the command gets stuck immediately after the TLS handshake.

* Expire in 50 ms for 1 (transfer 0x562c94210f50)
*   Trying 23.199.139.58...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x562c94210f50)
* Connected to www1.nseindia.com (23.199.139.58) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=IN; ST=Maharashtra; L=Mumbai; O=National Stock Exchange of India Ltd.; CN=www.nseindia.com
*  start date: Sep  2 00:00:00 2020 GMT
*  expire date: Dec 12 12:00:00 2020 GMT
*  subjectAltName: host "www1.nseindia.com" matched cert's "www1.nseindia.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=GeoTrust RSA CA 2018
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: www1.nseindia.com
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing

The tshark clearly indicates that the TLS handshake had completed, and the curl client did send the HTTP GET request, after which there is no response from the server.

Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens4'
    1 0.000000000   10.148.0.2 → 23.199.139.58 TCP 74 51830 → 443 [SYN] Seq=0 Win=65320 Len=0 MSS=1420 SACK_PERM=1 TSval=2896975156 TSecr=0 WS=128
    2 0.014462698 23.199.139.58 → 10.148.0.2   TCP 74 443 → 51830 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=979210844 TSecr=2896975156 WS=12
8
    3 0.014506462   10.148.0.2 → 23.199.139.58 TCP 66 51830 → 443 [ACK] Seq=1 Ack=1 Win=65408 Len=0 TSval=2896975170 TSecr=979210844
    4 0.015855789   10.148.0.2 → 23.199.139.58 TLSv1 583 Client Hello
    5 0.029133295 23.199.139.58 → 10.148.0.2   TCP 66 443 → 51830 [ACK] Seq=1 Ack=518 Win=30080 Len=0 TSval=979210859 TSecr=2896975171
    6 0.029490908 23.199.139.58 → 10.148.0.2   TLSv1.3 4162 Server Hello, Change Cipher Spec, Application Data
    7 0.029515497   10.148.0.2 → 23.199.139.58 TCP 66 51830 → 443 [ACK] Seq=518 Ack=4097 Win=62848 Len=0 TSval=2896975185 TSecr=979210859
    8 0.031222072 23.199.139.58 → 10.148.0.2   TCP 1474 443 → 51830 [ACK] Seq=4097 Ack=518 Win=30080 Len=1408 TSval=979210861 TSecr=2896975171 [TCP segment of a rea
ssembled PDU]
    9 0.031238234   10.148.0.2 → 23.199.139.58 TCP 66 51830 → 443 [ACK] Seq=518 Ack=5505 Win=64128 Len=0 TSval=2896975187 TSecr=979210861
   10 0.042769026 23.199.139.58 → 10.148.0.2   TLSv1.3 858 Application Data, Application Data, Application Data
   11 0.042797990   10.148.0.2 → 23.199.139.58 TCP 66 51830 → 443 [ACK] Seq=518 Ack=6297 Win=64128 Len=0 TSval=2896975198 TSecr=979210872
   12 0.043898975   10.148.0.2 → 23.199.139.58 TLSv1.3 146 Change Cipher Spec, Application Data
   13 0.044187939   10.148.0.2 → 23.199.139.58 TLSv1.3 169 Application Data
   14 0.057149099 23.199.139.58 → 10.148.0.2   TLSv1.3 353 Application Data
   15 0.057313736 23.199.139.58 → 10.148.0.2   TLSv1.3 353 Application Data
   16 0.057462731   10.148.0.2 → 23.199.139.58 TCP 66 51830 → 443 [ACK] Seq=701 Ack=6871 Win=64128 Len=0 TSval=2896975213 TSecr=979210887
   17 0.274408940   10.148.0.2 → 23.199.139.58 TCP 169 [TCP Retransmission] 51830 → 443 [PSH, ACK] Seq=598 Ack=6871 Win=64128 Len=103 TSval=2896975430 TSecr=9792108
87
   18 0.494346208   10.148.0.2 → 23.199.139.58 TCP 169 [TCP Retransmission] 51830 → 443 [PSH, ACK] Seq=598 Ack=6871 Win=64128 Len=103 TSval=2896975650 TSecr=9792108
87
   19 0.938332610   10.148.0.2 → 23.199.139.58 TCP 169 [TCP Retransmission] 51830 → 443 [PSH, ACK] Seq=598 Ack=6871 Win=64128 Len=103 TSval=2896976094 TSecr=9792108
87
   20 1.834328400   10.148.0.2 → 23.199.139.58 TCP 169 [TCP Retransmission] 51830 → 443 [PSH, ACK] Seq=598 Ack=6871 Win=64128 Len=103 TSval=2896976990 TSecr=9792108

After kill the curl client, I can also see a [FIN, ACK], but while the client is stuck there is no ingress response from the server.

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing

We can rule out the below logs which come from curl, because I have tried a GET on another site which support only HTTP1.1 with TLS1.2 and it worked and logged the same lines as well. This issue is only happening for this one site, i.e. https://www1.nseindia.com/ and is working fine for all the other sites which I have tried.

My questions are:

1. Why the communication channel is not working after the TLS handshake? If it was an issue with the firewall, my expectation was that the TLS channel would never get establised.
2. What is the root cause of the curl command to this site not working?
3. How can I get the curl command to this site working?

Note: This is working fine on every other non-google compute node which I have tested on.

I really hope this is just a firewall issue. Do ask for more details if that can help anyone answer this.

Answer 1

I have performed some curl test from different environments, but the result is always the same, the curl stuck without response.

I have tried from GCP (windows and linux), ubuntu PC, Windows PC, also ask for a test with a friend of mine far from my end, and the result was the same.

So, according to your first question; it makes me think that there is something on the url host, that after a TLS handshake is completed; "blocks" or "stops" the communication.

I'm not pretty sure if could it be the certificate or the server itself. If is it possible, could you please share a curl when it shows that the test was completed? Also it would be helpful if share the location of the test.

Your second and third question is that should it being working just performing the curl, let me review if I can think in another test or something that could help us to find the answers.

Answer 2

@blueboy1115 Thanks a lot for checking this. Based on your testing, I guess you're right, and the server is blocking the communication if the IP is not based probably in India. My google cloud instances are not from India, because the google cloud doesn't have any resources to allocate for a new instance in India.

This site is accessible on any browser and python scripts which I run on my laptop in India. But the same python scripts get stuck when I execute on the google VM instance hosted in America, Singapore and Sydney.

In the google instance:

• When I change the URL from https://www1.nseindia.com/ to https://www.nseindia.com/, then the request becomes HTTP2, and fails gracefully.

* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55e45a305f50)
> GET / HTTP/2
> Host: www.nseindia.com
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 403 
< server: AkamaiGHost
< mime-version: 1.0
< content-type: text/html
< content-length: 265
< expires: Thu, 01 Oct 2020 05:10:13 GMT
< date: Thu, 01 Oct 2020 05:10:13 GMT
< 
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
 
You don't have permission to access "http&#58;&#47;&#47;www&#46;nseindia&#46;com&#47;" on this server.<P>
Reference&#32;&#35;18&#46;5a0a0f17&#46;1601529013&#46;19fc9a7
</BODY>
</HTML>

-In my Windows laptop in India Interestingly on my Windows laptop on Indian ISP, where the site is working on the browser, and accessible on python scripts with the urllib library, the curl command fails for both www1.nseindia.com and www.nseindia.com after a timeout, and both attempt http1.1 because my Windows curl doesn't support HTTP2.

I have concluded that the server supports only HTTP2 and rejects IP address which are not hosted in India after the TLS handshake. This is the reason why it's working on my local laptop, via browser and via python with urllib.

If anyone has any other conclusions please do share.

Answer 3

From the results/logs posted it looks like the hosting server irrespective of the platform (gcp, etc..) the requests sent are being accepted / rejected basis on the region, generally this is done to prevent data/web scraping & this may be one of the security mechanism implemented at the remote server side(nseindia). That would be the reason curl -v https://www1.nseindia.com is going into the stale session after the TLS Handshake is done.

Answer 4

In my case I had to disable ipv6 on the proxying server, e.g. on Ubuntu:

sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1