Wait until all data is loaded from Webpage in HTTPoison

Question

I'm trying to load a webpage, where the data I'm interested in is loaded via Ajax after the initial page load. When I fetch the page, the text I currently see is Loading records....

I'm wondering what techniques I can use to wait until that section has loaded before the data is returned to me.

Currently I'm using the Elixir HTTPoison library, but am open to different techniques in Elixir/Erlang.

Answer 1

Libraries like HTTPoison only make the initial web request, returning the "server-rendered" content. They don't process the client-side code and markup. You need a headless browser / web-driver like PhantomJS, to actually process it and execute the javascript code (similar to how a normal web browser does).

Two popular Elixir libraries that do the job, (though they are mostly used for integration testing), are:

• Hound
• Wallaby

---

If the Ajax call is simple enough, another option is to first load the webpage and the scripts using HTTPoison, then extract the Javascript code using something like Floki, parse the Ajax call being made and then perform it manually via HTTPoison.