HTML Parser for response - Java

Question

Im using HttpClient to access a particualr website and the response i get is in the form of an HTML. Which parser or method I should use the parser the HTML and get what I want from the response. Note: Im using HttpClient with Java

Answer 1

Use jsoup.

jsoup is a Java library for working with real-world HTML. It provides a very convenient API for extracting and manipulating data, using the best of DOM, CSS, and jquery-like methods.

jsoup implements the WHATWG HTML5 specification, and parses HTML to the same DOM as modern browsers do.

• scrape and parse HTML from a URL, file, or string
• find and extract data, using DOM traversal or CSS selectors
• manipulate the HTML elements, attributes, and text
• clean user-submitted content against a safe white-list, to prevent XSS attacks
• output tidy HTML

jsoup is designed to deal with all varieties of HTML found in the wild; from pristine and validating, to invalid tag-soup; jsoup will create a sensible parse tree.

Answer 2

I would give htmlcleaner a try.

HTMLCleaner is Java library used to safely parse and transform any HTML found on web to well-formed XML. It is designed to be small, fast, flexible and independant. HtmlCleaner may be used in java code, as command line tool or as Ant task. Result of parsing is lightweight document object model which can easily be transformed to standards like DOM or JDom, or serialized to XML output in various ways (compact, pretty printed and so on).

You can use XPath with htmlcleaner to get contents within xml/html tags.Here is a nice
example Xpath Example

Answer 3

Sample code with jsoup and Java8:

// Imports:
...
import java.nio.charset.StandardCharsets;
import org.apache.commons.io.IOUtils;
...

// Execute the GET request:
...
HttpClient clientGet = HttpClientBuilder.create().build();
HttpGet get = new HttpGet(url);
HttpResponse res = clientGet.execute(get);

// Use jsoup to parse the html response:
// E.g. find all links with reference to myapp:
//  <a href="myapp">HelloWorldApp</a>
Document doc = Jsoup.parse(IOUtils.toString(res.getEntity().getContent(), StandardCharsets.UTF_8));
Elements links = doc.select("a[href~=myapp]");
for (Element link : links)
    String appName = link.html();
...