Docker-compose not working with localhost redirect

Question

I'm kinda new with docker and yarp so I probably made some basic mistake.

For context: I have a yarp gateway created in asp.net core and a micro-service also in asp.net core. When I go to the gateway url I want it to redirect to the micro-service. I was able to get this to work in two seperate containers.

However when I try to use docker-compose it wont work. The error I get the most is (http error 503) cannot assign requested address from my docker container.

This is my appsettings.json from the gateway. Yes there's stuff commented out. I've been toying with this for a while now.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "ReverseProxy": {
    "Routes": {
      "route1": {
        "ClusterId": "cluster1",
        "Match": {
          "Path": "{**catch-all}"
        },
        "Transforms": [
          { "PathPattern": "{**catch-all}" }
        ]
      }
      //"route2": {
      //  "ClusterId": "cluster2",
      //  "Match": {
      //    "Path": "/2/{**catch-all}"
      //  },
      //  "Transforms": [
      //    { "PathPattern": "{**catch-all}" }
      //  ]
      //}
    },
    "Clusters": {
      "cluster1": {
        "LoadBalancingPolicy": "RoundRobin",
        "Destinations": {
          "destination1": {
            "Address": "http://host.docker.internal:5100/weatherforecast"
          }
        }
      }
      //"cluster2": {
      //  "LoadBalancingPolicy": "RoundRobin",
      //  "Destinations": {
      //    "destination2": {
      //      "Address": "https://172.35.0.2:5101/PageCall"
      //    }
      //  }
      //}
    }
  }
}

this is my docker-compose

version: '3.4'

services:
  gateway:
    image: ${DOCKER_REGISTRY-}gateway
    build:
      context: .
      dockerfile: Gateway/Dockerfile
    #ports:
    #- "5000:5000"
    #- "5001:5001"
    networks:
     proxybackend:
  posting-service:
    image: ${DOCKER_REGISTRY-}postingservice
    build:
      context: .
      dockerfile: Posting-service/Dockerfile
    #ports:
    #- "5100:5100"
    #- "5101:5101"
    networks:
     proxybackend:

networks:
  proxybackend:
    name: proxybackend
    driver: bridge

and this is the docker-compose override

version: '3.4'

services:
  gateway:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=https://+:443;http://+:80
    ports:
      - "5000:80"   
      - "5001:443"
    volumes:
      - ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro
      - ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
  posting-service:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=https://+:443;http://+:80
    ports:
      - "5100:80"
      - "5101:443"
    volumes:
      - ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro
      - ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro

the full error from yarp is as follows

System.Net.Http.HttpRequestException: Cannot assign requested address (localhost:5101)
2024-03-12 10:06:02        ---> System.Net.Sockets.SocketException (99): Cannot assign requested address
2024-03-12 10:06:02          at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
2024-03-12 10:06:02          at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-03-12 10:06:02          at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
2024-03-12 10:06:02          at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
2024-03-12 10:06:02          --- End of inner exception stack trace ---
2024-03-12 10:06:02          at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
2024-03-12 10:06:02          at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-03-12 10:06:02          at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(HttpRequestMessage request)
2024-03-12 10:06:02          at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
2024-03-12 10:06:02          at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-03-12 10:06:02          at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2024-03-12 10:06:02          at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-03-12 10:06:02          at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)

I've tried everything I feel and I have no idea where it exactly goes wrong.

I've tried host.docker.internal, trying to do it with different IPs, I've used edge instead of chrome (as chrome will now always redirect to google no matter what I do) .