ConfigurationManager.GetConfigurationAsync fails to retrieve identity configuration in TestServer

Question

I'm using Microsoft.AspNetCore.TestHost version 2.2.0 and Microsoft.NET.Test.Sdk version 15.9.0. I'm using IdentityServer4 for OAuth2 provider for Jwt Bearer Authentication. The set up is working fine when deployed with real web servers. However, the issue occurs during integration tests using TestServer.

I have already registered HttpMessageHandler like:

private void InitializeServices<TStartup>(IServiceCollection services)
{
      var startupAssembly = typeof(TStartup).GetTypeInfo().Assembly;

      var applicationPartManager = new ApplicationPartManager();
      applicationPartManager.ApplicationParts.Add(new AssemblyPart(startupAssembly));
      applicationPartManager.FeatureProviders.Add(new ControllerFeatureProvider());
      applicationPartManager.FeatureProviders.Add(new ViewComponentFeatureProvider());

      services.AddSingleton(applicationPartManager);

      services.PostConfigure<IdentityServerAuthenticationOptions>(options =>
                                                                        {
                                                                            options.IntrospectionDiscoveryHandler = this.server.CreateHandler();
                                                                            options.IntrospectionBackChannelHandler = this.server.CreateHandler();
                                                                            options.JwtBackChannelHandler = this.server.CreateHandler();
                                                                        });
}

Where this.server is my test server. However, when I run my integration tests, I get the following error.

Result StackTrace:  
at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.JwtBearer.JwtBearerHandler.HandleAuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.AuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
   at IdentityServer4.AccessTokenValidation.IdentityServerAuthenticationHandler.HandleAuthenticateAsync() in C:\local\identity\server4\AccessTokenValidation\src\IdentityServer4.AccessTokenValidation\IdentityServerAuthenticationHandler.cs:line 61
   at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.AuthenticateAsync()
   at Microsoft.AspNetCore.Authentication.AuthenticationService.AuthenticateAsync(HttpContext context, String scheme)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
   at IdentityServer4.Hosting.BaseUrlMiddleware.Invoke(HttpContext context) in C:\local\identity\server4\IdentityServer4\src\IdentityServer4\Hosting\BaseUrlMiddleware.cs:line 36
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.TestHost.HttpContextBuilder.<>c__DisplayClass10_0.<<SendAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.TestHost.ClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at xxx.ClientControllerTest.CreateNewClientAsync() in xxx.ClientControllerTest.cs:line 167
   at xxx.ClientControllerTest.CreateNewClientTest() in xxx\ControllerTests\ClientControllerTest.cs:line 47
--- End of stack trace from previous location where exception was thrown ---
----- Inner Stack Trace -----
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.OpenIdConnect.OpenIdConnectConfigurationRetriever.GetAsync(String address, IDocumentRetriever retriever, CancellationToken cancel)
   at Microsoft.IdentityModel.Protocols.ConfigurationManager`1.GetConfigurationAsync(CancellationToken cancel)
----- Inner Stack Trace -----
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.IdentityModel.Protocols.HttpDocumentRetriever.GetDocumentAsync(String address, CancellationToken cancel)
----- Inner Stack Trace -----
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
Result Message: 
System.InvalidOperationException : IDX20803: Unable to obtain configuration from: '[PII is hidden by default. Set the 'ShowPII' flag in IdentityModelEventSource.cs to true to reveal it.]'.
---- System.IO.IOException : IDX20804: Unable to retrieve document from: '[PII is hidden by default. Set the 'ShowPII' flag in IdentityModelEventSource.cs to true to reveal it.]'.
-------- System.Net.Http.HttpRequestException : No connection could be made because the target machine actively refused it
------------ System.Net.Sockets.SocketException : No connection could be made because the target machine actively refused it