java.net.SocketException: Connection reset With HTTPConnection

Question

I am trying to hit some external API to fetch some data. When the data size is small, everything works fine but when the size of data returned by the API is big I get CONNECTION RESET exception. The below code is from java class InterfaceHelper and I have also marked the comment at the line no where I am getting exception [Its while trying to read data from InputStream].

I have tried to search so many question on STACKOVERFLOW itself but didn't find an appropriate answer. So please don't mark it as duplicate question. I want to know what is the reason behind this problem, and what is the proper solution for this problem. If you find this question as duplicate please answer it before marking it as duplicate. I dare you.

Find below my code which I have used. URL is some dummy url as for security reason I cannot mention the actual URL I have used.

 try{
        URL url = new URL("http://example.com/someParams/SOME-ACCESS-TOKEN");
        HttpURLConnection connection = (HttpURLConnection)url.openConnection();
        connection.setRequestMethod("GET");
        connection.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
        connection.setRequestProperty("Content-Language", "en-US");
        connection.setRequestProperty("X-EXAMPLE-LOGIN", "XXXXXXXX");
        connection.setRequestProperty("X-EXAMPLE-PASSWORD", "XXXXXX");
        connection.setUseCaches(false);
        connection.setDoInput(true);
        connection.setDoOutput(true);
        DataInputStream input = new DataInputStream(connection.getInputStream());
        String  ret = "";
        if(input!=null){
            for( int c = input.read(); c != -1; c = input.read() ) { //InterfaceHelper.java:695
                ret = ret + String.valueOf((char)c);
            }
        }

         if(input!=null)
             input.close();
         if(connection!=null)
             connection.disconnect();

                if(ret!=null && ret.length()>0){
                    return ret;
                }

    }catch(Exception e) {
        e.printStackTrace();
    } 

This the exception I get

        java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:196)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
        at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
        at sun.security.ssl.InputRecord.read(InputRecord.java:509)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
        at sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)
        at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:689)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3053)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3047)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3035)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at com.lsa.akosha.util.InterfaceHelper.hitApiBrandWatch(InterfaceHelper.java:695)
        at com.lsa.akosha.service.brand.BrandCronService.brandSentiments(BrandCronService.java:288)
        at com.lsa.akosha.util.thread.BrandWatchCronConverse.executeInternal(BrandWatchCronConverse.java:60)
        at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:113)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

Answer 1

I tried to debug it again and again, this time while debugging I found that reading data from DataInputStream is the culprit here.

The line dataInputStream.read() reads one char at a time, and I have to convert it into String. Below is the code

    String  ret = "";
    if(input!=null){
        for( int c = input.read(); c != -1; c = input.read() ) { //InterfaceHelper.java:695
            ret = ret + String.valueOf((char)c);
        }
    }

The main problem is that I am reading data into int and then convert it into char and then into String i.e. ret = ret + String.valueOf((char)c);.

Now I have removed the DataInputStream and used BufferInputStream and used bufferInputStream.readLine() which directly read's line in string from InputStream, so no type casting which saves lots of time in reading data from stream and hence till the time connection and stream expires it reads the data and then I close both.

This way my problem is resolved. I am posting my new code which helps me in solving my problem.

   try{
        URL url = new URL("http://example.com/someParams/SOME-ACCESS-TOKEN");
        HttpURLConnection connection = (HttpURLConnection)url.openConnection();
        connection.setRequestMethod("GET");
        connection.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
        connection.setRequestProperty("Content-Language", "en-US");
        connection.setRequestProperty("X-EXAMPLE-LOGIN", "XXXXXXXX");
        connection.setRequestProperty("X-EXAMPLE-PASSWORD", "XXXXXX");
        connection.setUseCaches(false);
        connection.setDoInput(true);
        connection.setDoOutput(true);
        connection.setConnectTimeout(99999999);
        connection.setReadTimeout(99999999);
        BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
        //DataInputStream input = new DataInputStream(connection.getInputStream());
        String  ret = "";
       /*   if(in!=null){
            for( int c = input.read(); c != -1; c = input.read() ) {
                ret = ret + String.valueOf((char)c);
                if(input==null || connection==null)
                    break;
            }
       }*/
       String inputLine;
       while ((inputLine = in.readLine()) != null) 
       {
            ret = ret + inputLine;
       }
       if(in!=null)
         in.close();
       if(connection!=null)
         connection.disconnect();
       if(ret!=null && ret.length()>0){
           return ret;
       }

   }catch(Exception e) {
      e.printStackTrace();
   }

Answer 2

maybe its just quirks, but I've found some "weird" things in your post.

1. You use SSL connection (as it appears in the stacktrace) but in the provided example this is no HTTPS at all.
2. application/x-www-form-urlencoded header is usually used with POST requests. Full explanation on this read here

I'm just saying that this can be misleading to some extent.

Now regarding the question itself.

All-in-all - connection reset means that for some reason the connection between the client and the server is broken. For example, the server decides to close the connection. The debugging process can be really tricky here, so if I were you, I would have tried a couple of techniques, depending on environment you're working on:

1. Remove SSL and work with plain HTTP. This way you can put a proxy in-between and analyze the network traffic better. Stuff like Burp can help here.

2. Try to eliminate the possibility that some firewall/proxy/whatever just dumps the connection for its own reasons. Maybe it worth to run this code in the same machine with the server (of course if its a viable option) just for the sake of testing how this works with "localhost", you know.

3. I'm not really familiar with this fairly low level API and know all its quirks. But maybe you can use HttpClient instead, this way you'll probably eliminate the need to know all the "low-level" flags, maybe its something wrong there.

Hope this helps

Answer 3

connection.setRequestMethod("GET");

Instead get method try using

connection.setRequestMethod("POST");

because get method has limit to send the data.

Answer 4

This is clearly not the real code, as noted in my comment above, but

connection.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
connection.setDoOutput(true);

You aren't sending any output so you shouldn't be calling these methods at all. And setDoOutput(true) contradicts setRequestMethod("GET");.

Alternatively, send some output.