How it is possible to use jodd.http.HttpRequest to load a page content that is generated by javascript?

Question

60 views Asked by papgyo At 13 January 2022 at 08:38

I try to load a page content with:

HttpResponse response2 = HttpRequest.get(_PAGE_URL).cookies(response.cookies()).send();

In a browser, the page source is full of javascript to generate the DOM, but in the Web Inspector of the browser I can see the generated source.

The question is, can I somehow retrieved the generated page content by Jodd's utilities?

There are 1 answers

**igr** · Accepted Answer · 2022-01-13 11:09:01

You can't. You can just download the static HTML content (as you did) and then you would need to render it using some other tool.

Since Java 8, you can use JavaFX's WebView Component (as far as I remember), but please search for other tools as well (maybe cef?)

EDIT See: https://github.com/igr/web-scraper (based on Selenium WebDriver). One thing I miss is better control over request/response.

There is also HtmlUnit, but from the reviews, it seems Selenium is a better choice.