How to make an HttpClient GetAsync wait for a webpage that loads data asynchronously?

Question

I'm making a snippet that sends data to a website that analyses it then sends back results.

Is there any way to make my GetAsych wait until the website finishes its calculation before getting a "full response"?

Ps: The await will not know if the page requested contains any asynchronous processing (eg: xhr calls)- I already use await and ReadAsByteArrayAsync()/ReadAsStringAsync()

Thank you!

Answer 1

What you're looking for here is the await operator. According to the docs:

The await operator suspends evaluation of the enclosing async method until the asynchronous operation represented by its operand completes.

Sample use within the context of an HttpClient object:

public static async Task Main()
{
    // send the HTTP GET request
    var response = await httpClient.GetAsync("my-url");

    // get the response string
    // there are other `ReadAs...()` methods if the return type is not a string
    var getResult = response.Content.ReadAsStringAsync();
}

Note that the method that encloses the await-ed code is marked as async and has a return type of Task (Task<T> would also work, depending on your needs).

Answer 2

You will need something like Selenium to not only fetch the HTML of the website but to fully render the page and execute any dynamic scripts.

You can then hook into some events, wait for certain DOM elements to appear or just wait some time until the page is fully initialized.

Afterwards you can use the API of Selenium to access the DOM and extract the information you need.

Example code:

using (var driver = new ChromeDriver(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location)))
    {
        driver.Navigate().GoToUrl(@"https://automatetheplanet.com/multiple-files-page-objects-item-templates/");
        var link = driver.FindElement(By.PartialLinkText("TFS Test API"));
        var jsToBeExecuted = $"window.scroll(0, {link.Location.Y});";
        ((IJavaScriptExecutor)driver).ExecuteScript(jsToBeExecuted);
        var wait = new WebDriverWait(driver, TimeSpan.FromMinutes(1));
        var clickableElement = wait.Until(ExpectedConditions.ElementToBeClickable(By.PartialLinkText("TFS Test API")));
        clickableElement.Click();
    }

Source: https://www.automatetheplanet.com/webdriver-dotnetcore2/