Java.io.IOException: Server returned HTTP response code: 520 for URL:

Question

I have a bit of Java code to download url data that is plagued by the error in the title. Sometimes it works, most time it fails. Has anyone come across this:

 URLConnection urlConnection = url2search.openConnection();
 urlConnection.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
 urlConnection.setRequestProperty("Accept","*/*");
 urlConnection.setDoInput(true);
 urlConnection.setDoOutput(false);

 try{
      reader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
     }catch(Exception r)
     {}

Now it fails consistently at the reader line with:

java.io.IOException: Server returned HTTP response code: 520 for URL:
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)

I can copy the url in to the search bar and it works fine. And as yet my web research on this topic has proved fruitless. Any suggestions?

Answer 1

An HTTP error code between 500 and 599 indicates a server failure. It could be at the requested document's origin server, or at a proxy server between the client and the origin server.

Code 520 itself is not documented by any of the HTTP specifications, so its specific meaning is unclear. If that code is being generated by a CloudFlare reverse proxy between your client and the origin server, however, then it signals a generic, unspecified connection error between the proxy and the origin server.

Any way around, the problem is basically external to your client. It may be that there is something about your request properties that tends to cause the server chain to fail as you observe it to do, but to debug it you need either to analyze the server logs and software, or else to reverse-engineer its behavior. If the problem is not exhibited in conjunction with your browser, then you could consider capturing the request/response involving your browser to see how it differs from the request/response involving your Java client.

Answer 2

Try bringing your user agent string up the latest.

See here: https://www.whatismybrowser.com/guides/the-latest-user-agent