Unable to use multiple audiences in OBO flow calling MS Graph

Question

I am trying to make an API that two separate client applications can use to get user data from GraphAPI. The two client applications have separate App Registrations in Azure. If I run the API with authentication for just one of the frontend application it works fine. However, when I try to configure it for both clients it just works for the one added last in Program.cs. For the other client calling the API I get an error:

"Assertion audience does not match the Client app presenting the assertion".

Any ideas?

This is a snippet from the Program.cs file:

builder.Services.AddAuthentication("AzureAd3001Scheme").AddMicrosoftIdentityWebApi(builder.Configuration, "AzureAd3001", jwtBearerScheme: "AzureAd3001Scheme")
.EnableTokenAcquisitionToCallDownstreamApi()
.AddMicrosoftGraph(builder.Configuration.GetSection("DownstreamAPI"))
.AddInMemoryTokenCaches();
builder.Services.AddAuthentication("AzureAd3000Scheme").AddMicrosoftIdentityWebApi(builder.Configuration, "AzureAd3000", jwtBearerScheme: "AzureAd3000Scheme")
.EnableTokenAcquisitionToCallDownstreamApi()
.AddMicrosoftGraph(builder.Configuration.GetSection("DownstreamAPI3000"))
.AddInMemoryTokenCaches();



builder.Services.AddAuthorization(options =>
{
    options.AddPolicy("AzureAd3000Policy", p =>
    {
        p.AuthenticationSchemes.Add("AzureAd3000Scheme");
        p.RequireAuthenticatedUser();
    });
    options.AddPolicy("AzureAd3001Policy", p =>
    {
        p.AuthenticationSchemes.Add("AzureAd3001Scheme");
        p.RequireAuthenticatedUser();
    });

Tried all kinds of solutions but it starting to doubt that the AddMicrosoftGraph part isn't able to handle multiple audiences.