Scrape web with "Load more" articles using goquery

57 views Asked by At

I have built a web scraper using goquery. However it can only retrieve the metadata of the first 14 or 15 articles, since the remaining articles can only be seen after manually clicking on a "Load more" button.

The new articles are not loaded asynchronously, because I can actually find their texts appearing under the "View page source" tab after they become visible (according to my limited knowledge on the "asynchronous" topic). So I guess that's a plus.

How can I deal with this problem? What are my options to scrape beyond the initial 15 articles?

0

There are 0 answers