How I get the html all source?

Question

I mean that I use below code to get html source from url. But it does not contain all source.Buffersize is problem or string size problem?

HttpURLConnection connection; 
            OutputStreamWriter request = null; 

                 URL url = null;    
                 String response = null;          
                 String parameters = "aranan="+et.getText();    

                 try 
                 { 
                     url = new URL("http://www.fragmanfan.com/arama.asp"); 
                     connection = (HttpURLConnection) url.openConnection(); 
                     connection.setDoOutput(true); 
                     connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded"); 

                     request = new OutputStreamWriter(connection.getOutputStream()); 
                     request.write(parameters); 
                     request.flush();             
                     String line = "";                
                     InputStreamReader isr = new InputStreamReader(connection.getInputStream()); 
                     BufferedReader reader = new BufferedReader(isr); 
                     StringBuilder sb = new StringBuilder(); 
                     while ((line = reader.readLine()) != null) 
                     { 
                         sb.append(line + "\n"); 
                     } 
                     // Response from server after login process will be stored in response variable.                 
                     response = sb.toString(); 
                     // You can perform UI operations here 
                     browser.loadDataWithBaseURL(null, response,"text/html", "UTF-8", null); 

                     isr.close(); 
                     reader.close(); 

                 } 
                 catch(IOException e) 
                 { 
                     // Error 
                 } 




        } 
    }); 

I try some things like BufferedReader reader = new BufferedReader(isr,8192); But it does not work.

Answer 1

create a WebRequest class. Than make your request and get response. I tried that website, it works.

WebRequest response = new WebRequest("http://www.fragmanfan.com/arama.asp?aranan=kurtlar", PostType.GET);
String htmltext = response.Get();
browser.loadDataWithBaseURL(null, htmltext, "text/html", "UTF-8", null);

WebRequest.class

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.UnknownHostException;
import java.nio.charset.Charset;
import org.apache.http.HttpResponse;
import org.apache.http.client.CookieStore;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.protocol.ClientContext;
import org.apache.http.impl.client.BasicCookieStore;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.protocol.BasicHttpContext;
import org.apache.http.protocol.HttpContext;

public class WebRequest {
  public enum PostType{
    GET, POST;
  }

  public String _url;
  public String response = "";
  public PostType _postType;
  CookieStore _cookieStore = new BasicCookieStore();

  public WebRequest(String url) {
    _url = url;
    _postType = PostType.POST;
  }

  public WebRequest(String url, CookieStore cookieStore) {
    _url = url;
    _cookieStore = cookieStore;
    _postType = PostType.POST;
  }

  public WebRequest(String url, PostType postType) {
    _url = url;
    _postType = postType;
  }

  public String Get() {
    HttpClient httpclient = new DefaultHttpClient();

    try {
      // Create local HTTP context
      HttpContext localContext = new BasicHttpContext();

      // Bind custom cookie store to the local context
      localContext.setAttribute(ClientContext.COOKIE_STORE, _cookieStore);

      HttpResponse httpresponse;
      if (_postType == PostType.POST)
      {
        HttpPost httppost = new HttpPost(_url);
        httpresponse = httpclient.execute(httppost, localContext);
      }
      else
      {
        HttpGet httpget = new HttpGet(_url);
        httpresponse = httpclient.execute(httpget, localContext);
      }

      StringBuilder responseString = inputStreamToString(httpresponse.getEntity().getContent());

      response = responseString.toString();
    }
    catch (UnknownHostException e) {
      e.printStackTrace();
    }
    catch (Exception e) {
      e.printStackTrace();
    }
    finally {
      // When HttpClient instance is no longer needed,
      // shut down the connection manager to ensure
      // immediate deallocation of all system resources
      httpclient.getConnectionManager().shutdown();
    }

    return response;
  }

  private StringBuilder inputStreamToString(InputStream is) throws IOException {
    String line = "";
    StringBuilder total = new StringBuilder();

    // Wrap a BufferedReader around the InputStream
    BufferedReader rd = new BufferedReader(new InputStreamReader(is,Charset.forName("iso-8859-9")));
    // Read response until the end
    while ((line = rd.readLine()) != null) {
      total.append(line);
    }

    // Return full string
    return total;
  }
}

Answer 2

I had some problem.

I used Log.i("tag", html) But logger has a max lenght of message. And my html text was clipped. There are two solutions:

1. split your html into small pieces
2. Make bigger size of max lenght of message as in this post: link