HttpUrlConnection returns ResponseCode 404 but any browser can access the URL anytime

Question

I need to open a connection to an URL address and to read html with StringBuffer. However when i try to establish connection i get file not found exception 404. When i write this address in any browser on Windows or Android, address can be accessed anytime. I do not understand why this happens or is there any workaround.

URL meteo = new URL("http://meteo.bg/bg/osnOpasni");
        HttpURLConnection connection = (HttpURLConnection) meteo.openConnection();
        connection.setRequestMethod("GET");
        connection.setRequestProperty("Connection", "keep-alive");
        connection.setRequestProperty("Accept", "text/html, */*");
        connection.setRequestProperty("X-Requested-With", "XMLHttpRequest");
        connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36");
        int responseCode = connection.getResponseCode();

If i use same code for any other URL - it works and the ResponseCode is 200. Is there anything i am missing???

Answer 1

I try de URL on a browser with DevTools and the 404 error appears in the first place using the F12 Network analisis:

Seems like the URL is answering with a 404 error:

No error on your code the error is there in the website.

Answer 2

I was able to execute Http request to your URL and got the content without any problem. Here is my code:

public static void simpleHttpTest() {
    HttpClient client = new HttpClient();
    ResponseHolder<String> responseHolder;
    try {
        responseHolder = client.setRequestHeader("Accept", "text/html")
            .sendHttpRequest("http://meteo.bg/bg/osnOpasni", HttpClient.HttpMethod.GET);
        System.out.println("Response received:\n" + responseHolder.getResponseContent());
    } catch (HttpClientCommunicationException hcce) {
        responseHolder = hcce.getResponseHolder();
        System.out.println("Error occurred:\n" + responseHolder.getResponseCode() + " " + responseHolder.getResponseMessage()
            + "\nResponse Headers: " + responseHolder.getResponseHeaders());
    }
}

Class HttpClient is a 3d party Http Client, and in general, I would suggest that you use some 3d party Http client and there are many available. This particular HttpClient comes with MgntUtils Open Source library written and maintained by me. This HTTP client is very simple to use, but it does not provide all the width of the functionality provided by other HTTP clients. Here is Javadoc for HttpClient class that I used. The MgntUtils library can be obtained as Maven artifact or from GitHub (with source code and Javadoc). Also If you want to use well known Http clients I would suggest trying:

1. Apache Http Client
2. OK Http Client
   
   BTW, the reason that you were getting 404 error may have been due to the site temporarily unavailable.

Answer 3

I have found a pretty simple solution of the problem with the error on html page which i am connecting to by using getErrorStream(). Here is the working code for this case:

URL meteo = new URL("http://meteo.bg/bg/osnOpasni");
        HttpURLConnection connection = (HttpURLConnection) meteo.openConnection();
        BufferedReader in = new BufferedReader(new InputStreamReader(connection.getErrorStream()));

i hope this will help someone else, who has to deal with errors on html pages.