Ambassador mapping causing permanent redirects

Question

Been trying to diagnose an issue with a couple ambassador mappings which appear to result in a 301 Permanent Redirect when from what I understand it shouldn't.

Here is the behavior:

1. Go to https://[api domain]/pubsub/[random string]
2. Redirected to https://[api domain]/[same random string] with 404 response (expected from main api service)
3. Go to https://[api domain]/pubsub/[same random string] again
4. Redirected with 301 to https://[api domain]/[same random string]

API Mapping

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: api
spec:
  prefix: /
  service: api
  regex_rewrite:
    pattern: ^/docs/$
    substitution: /docs/index.html
  host: [api domain]
  timeout_ms: 60000

PubSub Mapping

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: pubsub-api
spec:
  prefix: /pubsub
  service: pubsub
  host: [api domain]
  timeout_ms: 60000
  allow_upgrade:
    - websocket
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: pubsub-edge
spec:
  prefix: /
  service: pubsub
  host: [deprecated websocket domain]
  timeout_ms: 60000
  allow_upgrade:
    - websocket

This does NOT happen when using the deprecated websocket domain but appears to be some sort of conflict with the two mappings on the api domain.

I should note that there are zero 301 redirects in my code so this appears to be coming from emissary itself and that the issue happens with any path so long as it starts with /pubsub

My theory was that the issue was caused by the ordering of my deployments in Kustomize but changing the order to add the api mapping after the pubsub mapping seemed to only make the issue worse, resulting in blank responses for valid requests to the api service.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - api
  - graphql
  - pubsub

Answer 1

prefix: /pubsub? Just to be safe, check that the prefix for the pubsub-api mapping ends with a / to avoid conflicts:

prefix: /pubsub/

Mappings in Ambassador are evaluated in order, and the first match will be used. You should make sure more specific mappings are placed before more general ones.

See more at Advanced Mapping configuration / Mapping evaluation order.

You might want to make sure the mappings are applied in the correct order, which could be influenced by how resources are organized in your Kustomize setup.

You could try and define each mapping in its own file. That makes managing their order easier.
For example, have pubsub-api-mapping.yaml and api-mapping.yaml.
In your Kustomize configuration, explicitly list resources in the order you want them to be applied. Kustomize applies resources in the order they are listed in the resources block of the kustomization.yaml file.

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
    - pubsub-api-mapping.yaml
    - api-mapping.yaml
    - other-resources...

Use kubectl apply -k path/to/kustomize/directory to apply the Kustomization configuration. That will apply the resources in the order they are listed.
After applying, you can check the order in which the mappings were applied using kubectl get mappings -o yaml.

---

I had already tried a different order in my kustomization. Doing this only made the issue worse, causing blank responses.

Given that changing the order in your Kustomize configuration did not resolve the issue and even worsened it, check if applying resources in different namespaces can implicitly manage the order. If applicable, you could try deploying the pubsub and api services in separate namespaces. (mentioned in this answer and in the documentation)

Also, an init container in Kubernetes runs to completion before the main container starts. That can be used to enforce ordering. For instance, an init container could run a script that checks for the existence of the pubsub-api mapping and only allows the main application to start once it is confirmed.

As a temporary measure, for debugging, you might manually apply the configurations in steps, first deploying the pubsub-api mapping and verifying its proper function before deploying the api mapping.

Check the:

• readiness probes to your services to make sure they are fully functional before they start receiving traffic.
• logs for both the Ambassador instance and the backend services. The logs might provide clues on why the redirection or blank responses are happening.