How to overwrite appsettings.json in WebApplicationFactory?

Question

I want to test my webapis middleware. I'm using a

public class CustomWebApplicationFactory : WebApplicationFactory<Program> { }

My test looks like this

[Fact]
public async Task WhenCalledUnauthenticated_ReturnsException()
{
    // Arrange
    var dto = new DeleteDefaultSaleSettingRequestDto { Id = 1 };

    this._client = this._factory.WithWebHostBuilder(builder =>
        {
           var projectDir = Directory.GetCurrentDirectory();
           var configPath = Path.Combine(projectDir, "appsettings.json");
                
           builder.ConfigureAppConfiguration((context,conf) =>
           {
               conf.AddJsonFile(configPath);
           });
                
           builder.ConfigureServices(services =>
           {
               var mediator = Substitute.For<IMediator>();
               mediator.Send(Arg.Any<DeleteDefaultSaleSettingCommand>(), Arg.Any<CancellationToken>())
                        .Returns(Result<bool?>.SuccessResult(true));
                    services.AddSingleton(mediator);
                    
                });
            })
            .CreateClient();

        var queryStringHelper = new QueryStringHelper("/api/ChannelSetting/DeleteDefaultSaleSetting");
        var uri = queryStringHelper.AddQueryString(dto);

        // Act
        var response = await this._client.DeleteAsync(uri);

        // Assert
        Assert.NotNull(response);
        Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode);
    }

The problem is that in my program I load appsetting.json (from the WepApiFolder, not the test folder) to be used for Serilog. These settings contains a setting for MS sql server and this breaks the test. So I'm trying to overload the appsetting.json in my tests but I suspect my loading happens after bootstrapping program.

Program.cs

builder.Host.UseSerilog((context, configuration) =>
 configuration.ReadFrom.Configuration(context.Configuration));