Using AWS App Mesh for EKS with a shared mesh

Question

0All of the AWS AppMesh examples currently in the examples repo assume that you have a mesh with the same namespace name as the EKS namespace. See here: https://github.com/aws/aws-app-mesh-examples/blob/main/walkthroughs/howto-k8s-grpc/v1beta2/manifest.yaml.template

What I'm now trying to do is create an AWS AppMesh which spans the whole cluster. So I've created an Mesh called "development" and removed the namespace selector:

apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: | redacted
  creationTimestamp: "2023-12-06T23:03:02Z"
  finalizers:
  - finalizers.appmesh.k8s.aws/mesh-members
  - finalizers.appmesh.k8s.aws/aws-appmesh-resources
  generation: 3
  labels:
    app.kubernetes.io/managed-by: pulumi
  name: development
  resourceVersion: "92450517"
  uid: b1d743cd-dbdf-47cc-99bb-da43ebd653b3
spec:
  awsName: development
  namespaceSelector: {}

This has created the mesh fine. I'm then trying to deploy the GRPC example, but in a different namespace:

---
apiVersion: v1
kind: Namespace
metadata:
  name: grpc
  labels:
    mesh: grpc
    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  name: development
spec:
  namespaceSelector:
    matchLabels:
      mesh: development
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: client
  namespace: grpc
spec:
  podSelector:
    matchLabels:
      app: client
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  backends:
    - virtualService:
        virtualServiceRef:
          name: color
  serviceDiscovery:
    awsCloudMap:
      namespaceName: howto-k8s-grpc.svc.cluster.local
      serviceName: client
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualNode
metadata:
  name: server
  namespace: grpc
spec:
  podSelector:
    matchLabels:
      app: color
      version: server
  listeners:
    - portMapping:
        port: 8080
        protocol: grpc
      healthCheck:
        port: 8080
        protocol: grpc
        healthyThreshold: 2
        unhealthyThreshold: 3
        timeoutMillis: 2000
        intervalMillis: 5000
  serviceDiscovery:
    awsCloudMap:
      namespaceName: howto-k8s-grpc.svc.cluster.local
      serviceName: color
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualService
metadata:
  name: color
  namespace: grpc
spec:
  awsName: color.howto-k8s-grpc.svc.cluster.local
  provider:
    virtualRouter:
      virtualRouterRef:
        name: color
---
apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: color
  namespace: grpc
spec:
  listeners:
    - portMapping:
        port: 8080
        protocol: grpc
  routes:
    - name: route
      grpcRoute:
        match:
          serviceName: color.ColorService
          methodName: GetColor
        action:
          weightedTargets:
            - virtualNodeRef:
                name: server
              weight: 1
---
# Service per VirtualNode is a no-op when using CloudMap
apiVersion: v1
kind: Service
metadata:
  name: client
  namespace: grpc
spec:
  ports:
    - port: 8080
      name: http
  selector:
    app: client
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: client
  namespace: grpc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      labels:
        app: client
    spec:
      containers:
        - name: app
          image: 186241287477.dkr.ecr.us-east-1.amazonaws.com/howto-k8s-grpc/color_client
          ports:
            - containerPort: 8080
          env:
            - name: "COLOR_HOST"
              value: "color.howto-k8s-grpc.svc.cluster.local:8080"
            - name: "PORT"
              value: "8080"
---
# Service per VirtualNode is a no-op when using CloudMap
apiVersion: v1
kind: Service
metadata:
  name: server
  namespace: grpc
spec:
  ports:
    - port: 8080
      name: http
  selector:
    app: color
    version: server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: server
  namespace: grpc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: color
      version: server
  template:
    metadata:
      labels:
        app: color
        version: server
    spec:
      containers:
        - name: app
          image: 186241287477.dkr.ecr.us-east-1.amazonaws.com/howto-k8s-grpc/color_server
          ports:
            - containerPort: 8080
          env:
            - name: "COLOR"
              value: "no color!"
            - name: "PORT"
              value: "8080"
---
apiVersion: v1
kind: Service
metadata:
  name: color
  namespace: grpc
spec:
  ports:
    - port: 8080
      name: http
  selector:
    app: color

This is failing to work when I call the color service, with:

$ curl localhost:8080/getColor
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp: lookup <redacted>.development.svc.cluster.local on 10.100.0.10:53: no such host"

Is it even possible for this to work? If so, how?