Polly: how to combine TimeoutPolicy and RetryPolicy to request a Func

Question

I'm trying to combine TimeoutPolicy and RetryPolicy for a API call done in a Func, but I don't found a way to achieve this.

If I only use the RetryPolicy, it's working fine.

I've a GetRequest method that call the HttpClient and return the datas:

async Task<Data> GetRequest(string api, int id)
{
    var httpClient = new HttpClient();
    var response = await httpClient.GetAsync($"{api}{id}");

    var rawResponse = await response.Content.ReadAsStringAsync();
    return JsonConvert.DeserializeObject<Data>(rawResponse);
}

I also have the Func that will embed the call to this method: var func = new Func<Task<Data>>(() => GetRequest(api, i));

I call the service like this: Results.Add(await _networkService.RetryWithoutTimeout<Data>(func, 3, OnRetry));

This RetryWithoutTimeout method is like this:

async Task<T> RetryWithoutTimeout<T>(Func<Task<T>> func, int retryCount = 1, Func<Exception, int, Task> onRetry = null)
{
    var onRetryInner = new Func<Exception, int, Task>((e, i) =>
    {
        return Task.Factory.StartNew(() => {
#if DEBUG
            System.Diagnostics.Debug.WriteLine($"Retry #{i} due to exception '{(e.InnerException ?? e).Message}'");
#endif
        });
    });

    return await Policy.Handle<Exception>()
                        .RetryAsync(retryCount, onRetry ?? onRetryInner)
                        .ExecuteAsync<T>(func);
}

I've updated this code to use a TimeoutPolicy, with a new RetryWithTimeout method:

async Task<T> RetryWithTimeout<T>(Func<Task<T>> func, int retryCount = 1, Func<Exception, int, Task> onRetry = null, int timeoutDelay = 30)
{
    var onRetryInner = new Func<Exception, int, Task>((e, i) =>
    {
        return Task.Factory.StartNew(() => {
#if DEBUG
            System.Diagnostics.Debug.WriteLine($"Retry #{i} due to exception '{(e.InnerException ?? e).Message}'");
#endif
        });
    });

    var retryPolicy = Policy
        .Handle<Exception>()
        .RetryAsync(retryCount, onRetry ?? onRetryInner);

    var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromSeconds(timeoutDelay));

    var policyWrap = timeoutPolicy.WrapAsync((IAsyncPolicy)retryPolicy);

    return await policyWrap.ExecuteAsync(
                    async ct => await Task.Run(func),
                    CancellationToken.None
                    );
}

But I don't see how to manage the GetRequest() method: all my tests have failed...

Edit: I've created a sample based on the @Peter Csala comment.

So first, I've just updated the number of retries to check if the retryPolicy was correctly applied:

private const int TimeoutInMilliseconds = 2500;
private const int MaxRetries = 3;
private static int _times;

static async Task Main(string[] args)
{
    try
    {
        await RetryWithTimeout(TestStrategy, MaxRetries);
    }
    catch (Exception ex)
    {
        WriteLine($"{nameof(Main)} - Exception - Failed due to: {ex.Message}");
    }
    Console.ReadKey();
}

private static async Task<string> TestStrategy(CancellationToken ct)
{
    WriteLine($"{nameof(TestStrategy)} has been called for the {_times++}th times.");
    await Task.Delay(TimeoutInMilliseconds * 2, ct);
    return "Finished";
}

internal static async Task<T> RetryWithTimeout<T>(Func<CancellationToken, Task<T>> func, int retryCount = 1, Func<Exception, int, Task> onRetry = null, int timeoutDelay = TimeoutInMilliseconds)
{
    WriteLine($"NetworkService - {nameof(RetryWithTimeout)}");
    var onRetryInner = new Func<Exception, int, Task>((e, i) =>
    {
        WriteLine($"NetworkService - {nameof(RetryWithTimeout)} #{i} due to exception '{(e.InnerException ?? e).Message}'");
        return Task.CompletedTask;
    });

    var retryPolicy = Policy
        .Handle<Exception>()
        .RetryAsync(retryCount, onRetry ?? onRetryInner);

    var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromMilliseconds(timeoutDelay));

    var policyWrap = Policy.WrapAsync(retryPolicy, timeoutPolicy); //Important part #1

    return await policyWrap.ExecuteAsync(
                    async ct => await func(ct), //Important part #2
                    CancellationToken.None);
}

Regarding the logs, it's well the case:

NetworkService - RetryWithTimeout
TestStrategy has been called for the 0th times.
NetworkService - RetryWithTimeout - Retry #1 due to exception 'A task was canceled.'
TestStrategy has been called for the 1th times.
NetworkService - RetryWithTimeout - Retry #2 due to exception 'A task was canceled.'
TestStrategy has been called for the 2th times.
NetworkService - RetryWithTimeout - Retry #3 due to exception 'A task was canceled.'
TestStrategy has been called for the 3th times.
Main - TimeoutRejectedException - Failed due to: The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.

Then, I've changed the policyWrap as I need a global timeout:

private static async Task<string> TestStrategy(CancellationToken ct)
{
    WriteLine($"{nameof(TestStrategy)} has been called for the {_times++}th times.");
    await Task.Delay(1500, ct);
    throw new Exception("simulate Exception");
}

var policyWrap = timeoutPolicy.WrapAsync(retryPolicy);

Regarding the logs, it's also correct:

TestStrategy has been called for the 0th times.
NetworkService - RetryWithTimeout #1 due to exception 'simulate Exception'
TestStrategy has been called for the 1th times.
NetworkService - RetryWithTimeout #2 due to exception 'A task was canceled.'
Main - TimeoutRejectedException - Failed due to: The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.

After that, I've implemented a method that call an API, with some Exceptions, to be more close to my need:

static async Task Main(string[] args)
{
    try
    {
        await RetryWithTimeout(GetClientAsync, MaxRetries);
    }
    catch (TimeoutRejectedException trEx)
    {
        WriteLine($"{nameof(Main)} - TimeoutRejectedException - Failed due to: {trEx.Message}");
    }
    catch (WebException wEx)
    {
        WriteLine($"{nameof(Main)} - WebException - Failed due to: {wEx.Message}");
    }
    catch (Exception ex)
    {
        WriteLine($"{nameof(Main)} - Exception - Failed due to: {ex.Message}");
    }
    Console.ReadKey();
}

private static async Task<CountriesResponse> GetClientAsync(CancellationToken ct)
{
    WriteLine($"{nameof(GetClientAsync)} has been called for the {_times++}th times.");
    HttpClient _client = new HttpClient();
    try
    {
        var response = await _client.GetAsync(apiUri, ct);
        // ! The server response is faked through a Proxy and returns 500 answer !
        if (!response.IsSuccessStatusCode)
        {
            WriteLine($"{nameof(GetClientAsync)} - !response.IsSuccessStatusCode");
            throw new WebException($"No success status code {response.StatusCode}");
        }
        var rawResponse = await response.Content.ReadAsStringAsync();
        WriteLine($"{nameof(GetClientAsync)} - Finished");
        return JsonConvert.DeserializeObject<CountriesResponse>(rawResponse);
    }
    catch (TimeoutRejectedException trEx)
    {
        WriteLine($"{nameof(GetClientAsync)} - TimeoutRejectedException : {trEx.Message}");
        throw trEx;
    }
    catch (WebException wEx)
    {
        WriteLine($"{nameof(GetClientAsync)} - WebException: {wEx.Message}");
        throw wEx;
    }
    catch (Exception ex)
    {
        WriteLine($"{nameof(GetClientAsync)} - other exception: {ex.Message}");
        throw ex;
    }
}

The logs are still correct:

NetworkService - RetryWithTimeout
GetClientAsync has been called for the 0th times.
GetClientAsync - !response.IsSuccessStatusCode
GetClientAsync - WebException: No success status code InternalServerError
NetworkService - RetryWithTimeout #1 due to exception 'No success status code InternalServerError'
GetClientAsync has been called for the 1th times.
GetClientAsync - !response.IsSuccessStatusCode
GetClientAsync - WebException: No success status code InternalServerError
NetworkService - RetryWithTimeout #2 due to exception 'No success status code InternalServerError'
GetClientAsync has been called for the 2th times.
GetClientAsync - !response.IsSuccessStatusCode
GetClientAsync - WebException: No success status code InternalServerError
NetworkService - RetryWithTimeout #3 due to exception 'No success status code InternalServerError'
GetClientAsync has been called for the 3th times.
GetClientAsync - other exception: The operation was canceled.
Main - TimeoutRejectedException - Failed due to: The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.

Finally, I would like to be able to call a "generic" method, that I could reuse for each API call. This method will be like this:

static async Task<T> ProcessGetRequest<T>(Uri uri, CancellationToken ct)
{
    WriteLine("ApiService - ProcessGetRequest()");

    HttpClient _client = new HttpClient();

    var response = await _client.GetAsync(uri);
    if (!response.IsSuccessStatusCode)
    {
        WriteLine("ApiService - ProcessGetRequest() - !response.IsSuccessStatusCode");
        throw new WebException($"No success status code {response.StatusCode}");
    }
    var rawResponse = await response.Content.ReadAsStringAsync();

    return JsonConvert.DeserializeObject<T>(rawResponse);
}

But for this, I have to pass at the same time the CancellationToken and the Api Uri through the the RetryWithTimeout and I don't see how to manage this.

I've tried to change the signature of RetryWithTimeout by something like:

internal static async Task<T> RetryWithTimeout<T>(Func<Uri, CancellationToken, Task<T>> func, int retryCount = 1, Func<Exception, int, Task> onRetry = null, int timeoutDelay = TimeoutInMilliseconds)

But I don't find how to the manage the Func...

Would you have an idea or an explanation?

Answer 1

I finally found a solution that works, this solution has been completed by @Peter Csala.

private const int TimeoutInMilliseconds = 2500;
private const int MaxRetries = 3;

private static Uri apiUri = new Uri("https://api/param");
private static int _times;

public static Country[] Countries
{
    get;
    set;
}

static async Task Main(string[] args)
{
    try
    {
        await LoadCountriesWithRetry(false);
    }
    catch (Exception ex)
    {
        WriteLine($"{nameof(Main)} - Exception - Failed due to: {ex.Message}");
    }
    Console.ReadKey();
}

static async Task LoadCountriesWithRetry(bool shouldWaitAndRetry)
{
    WriteLine($"{nameof(LoadCountriesWithRetry)}");
    try
    {
        Countries = await GetCountriesWithRetry();
    }
    catch (TimeoutRejectedException trE)
    {
        WriteLine($"{nameof(LoadCountriesWithRetry)} - TimeoutRejectedException : {trE.Message}");
    }   
    catch (WebException wE)
    {
        WriteLine($"{nameof(LoadCountriesWithRetry)} - WebException : {wE.Message}");
    }
    catch (Exception e)
    {
        WriteLine($"{nameof(LoadCountriesWithRetry)} - Exception : {e.Message}");
    }
}

public static async Task<Country[]> GetCountriesWithRetry()
{
    WriteLine($"{nameof(GetCountriesWithRetry)}");
    var response = await GetAndRetry<CountriesResponse>(uri, MaxRetries);
    return response?.Countries;
}

static Func<CancellationToken, Task<T>> IssueRequest<T>(Uri uri) => ct => ProcessGetRequest<T>(ct, uri);

public static async Task<T> GetAndRetry<T>(Uri uri, int retryCount, Func<Exception, int, Task> onRetry = null)
    where T : class
{
    WriteLine($"{nameof(GetAndRetry)}");
    return await RetryWithTimeout<T>(IssueRequest<T>(uri), retryCount);
}

static async Task<T> ProcessGetRequest<T>(CancellationToken ct, Uri uri)
{
    WriteLine($"{nameof(ProcessGetRequest)}");
    HttpClient _client = new HttpClient();
    var response = await _client.GetAsync(uri, ct);
    if (!response.IsSuccessStatusCode)
    {
        WriteLine($"{nameof(ProcessGetRequest)} - !response.IsSuccessStatusCode");
        throw new WebException($"No success status code {response.StatusCode}");
    }
    var rawResponse = await response.Content.ReadAsStringAsync();
    WriteLine($"{nameof(ProcessGetRequest)} - Success");
    return JsonConvert.DeserializeObject<T>(rawResponse);
}

internal static async Task<T> RetryWithTimeout<T>(Func<CancellationToken, Task<T>> func, Uri uri, int retryCount = 1, Func<Exception, int, Task> onRetry = null, int timeoutDelay = TimeoutInMilliseconds)
{
    WriteLine($"{nameof(RetryWithTimeout)}");
    var onRetryInner = new Func<Exception, int, Task>((e, i) =>
    {
        WriteLine($"{nameof(RetryWithTimeout)} - onRetryInner #{i} due to exception '{(e.InnerException ?? e).Message}'");
        return Task.CompletedTask;
    });

    var retryPolicy = Policy
        .Handle<Exception>()
        .RetryAsync(retryCount, onRetry ?? onRetryInner);

    var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromMilliseconds(timeoutDelay));

    var policyWrap = timeoutPolicy.WrapAsync(retryPolicy);

    return await policyWrap.ExecuteAsync(
                    async (ct) => await func(ct),
                    CancellationToken.None);
}

I get these results:

GetCountriesWithRetry
GetAndRetry
RetryWithTimeout
ProcessGetRequest
ProcessGetRequest - !response.IsSuccessStatusCode
RetryWithTimeout - onRetryInner #1 due to exception 'No success status code InternalServerError'
ProcessGetRequest
ProcessGetRequest - !response.IsSuccessStatusCode
RetryWithTimeout - onRetryInner #2 due to exception 'No success status code InternalServerError'
LoadCountriesWithRetry - TimeoutRejectedException : The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.

=> the API calls are retried until there is the Timeout

Thank you @Peter Csala!

Answer 2

You need to pass the CancellationToken to the to-be-cancelled (due to timeout) function.

So, let's suppose you have the following simplified method:

private const int TimeoutInMilliseconds = 1000;
private static int _times;
private static async Task<string> TestStrategy(CancellationToken ct)
{
    Console.WriteLine($"{nameof(TestStrategy)} has been called for the {_times++}th times.");
    await Task.Delay(TimeoutInMilliseconds * 2, ct);
    return "Finished";
}

So, your RetryWithTimeout could be adjusted / amended like this:

static async Task<T> RetryWithTimeout<T>(Func<CancellationToken, Task<T>> func, int retryCount = 1, Func<Exception, int, Task> onRetry = null, int timeoutDelay = TimeoutInMilliseconds)
{
    var onRetryInner = new Func<Exception, int, Task>((e, i) =>
    {
        Console.WriteLine($"Retry #{i} due to exception '{(e.InnerException ?? e).Message}'");
        return Task.CompletedTask;
    });

    var retryPolicy = Policy
        .Handle<Exception>()
        .RetryAsync(retryCount, onRetry ?? onRetryInner);

    var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromMilliseconds(timeoutDelay));

    var policyWrap = Policy.WrapAsync(retryPolicy, timeoutPolicy); //Important part #1

    return await policyWrap.ExecuteAsync(
                    async ct => await func(ct), //Important part #2
                    CancellationToken.None);
}

Important part #1 - Retry is the outer, Timeout is the inner policy
Important part #2 - CancellationToken is passed to the to-be-cancelled function due to timeout

The following usage

static async Task Main(string[] args)
{
    try
    {
        await RetryWithTimeout(TestStrategy);
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Failed due to: {ex.Message}");
    }

    Console.ReadKey();
}

will produce the following output:

TestStrategy has been called for the 0th times.
Retry #1 due to exception 'A task was canceled.'
TestStrategy has been called for the 1th times.
Failed due to: The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.

Please bear in mind that there is 0th attempt before retry kicks in.