Using an AuthorizationPolicy causes a 503 error

Question

When using an AuthorizationPolicy, I came across a 503 error trying to access the application. The error seems to be between the gateway and the service, but traffic never reaches the service. For instance, when using this policy to allow all traffic:

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: authorize-allow-all
  namespace: dagster
spec:
  selector:
      matchLabels:
        component: dagit
  action: ALLOW

Logs showing 503 error:

022-03-23T15:25:47.112544Z  debug   envoy filter    tls inspector: new connection accepted
2022-03-23T15:25:47.117785Z debug   envoy filter    tls:onServerName(), requestedServerName: radacted
2022-03-23T15:25:47.117863Z debug   envoy conn_handler  [C8274] new connection
2022-03-23T15:25:47.202884Z debug   envoy http  [C8274] new stream
2022-03-23T15:25:47.202965Z debug   envoy http  [C8274][S2813519150144299458] request headers complete (end_stream=false):
‘:authority’, ‘radacted’
‘:path’, ‘/graphql’
‘:method’, ‘POST’
‘dagster-testing’, ‘3’
‘authorization’, ‘Bearer eyJ0...’
‘content-type’, ‘application/json’
‘user-agent’, ‘PostmanRuntime/7.29.0’
‘accept’, ‘*/*’
‘cache-control’, ‘no-cache’
‘postman-token’, ‘d625526b-3f8a-4f9f-9b8d-352a1683b9d5’
‘accept-encoding’, ‘gzip, deflate, br’
‘connection’, ‘keep-alive’
‘content-length’, ‘110’
‘cookie’, ‘_oauth2_proxy_csrf=radacted’
2022-03-23T15:25:47.203065Z debug   envoy router    [C8274][S2813519150144299458] cluster ’outbound|3000||dagster-dagit.dagster.svc.cluster.local’ match for URL ‘/graphql’
2022-03-23T15:25:47.203121Z debug   envoy router    [C8274][S2813519150144299458] router decoding headers:
‘:authority’, ‘radacted’
‘:path’, ‘/graphql’
‘:method’, ‘POST’
‘:scheme’, ‘https’
‘dagster-testing’, ‘3’
‘authorization’, ‘Bearer eyJ0...’
‘content-type’, ‘application/json’
‘user-agent’, ‘PostmanRuntime/7.29.0’
‘accept’, ‘*/*’
‘cache-control’, ‘no-cache’
‘postman-token’, ‘d625526b-3f8a-4f9f-9b8d-352a1683b9d5’
‘accept-encoding’, ‘gzip, deflate, br’
‘content-length’, ‘110’
‘cookie’, ‘_oauth2_proxy_csrf=radacted’
‘x-forwarded-for’, ‘10.11.225.213’
‘x-forwarded-proto’, ‘https’
‘x-envoy-internal’, ‘true’
‘x-request-id’, ‘8aee0b05-65b3-420b-9ea4-2c3c11f3f93b’
‘x-envoy-decorator-operation’, ‘dagster-dagit.dagster.svc.cluster.local:3000/*’
‘x-envoy-peer-metadata’, ‘ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKGQoNSVNUSU9fVkVSU0lPThIIGgYxLjEwLjAK0AUKBkxBQkVMUxLFBSrCBQoXCgNhcHASEBoOaXN0aW8tb3BlcmF0b3IKKAobYXBwLmt1YmVybmV0ZXMuaW8vY29tcG9uZW50EgkaB2luZ3Jlc3MKJQobYXBwLmt1YmVybmV0ZXMuaW8vbWFuYWdlZEJ5EgYaBEhlbG0KMgoWYXBwLmt1YmVybmV0ZXMuaW8vbmFtZRIYGhZpc3Rpby1vcGVyYXRvci1pbmdyZXNzCi0KGWFwcC5rdWJlcm5ldGVzLmlvL3BhcnQtb2YSEBoOaXN0aW8tb3BlcmF0b3IKJQoZYXBwLmt1YmVybmV0ZXMuaW8vdmVyc2lvbhIIGgZ2MC4wLjIKEwoFY2hhcnQSChoIZ2F0ZXdheXMKHQoNaGVsbS5zaC9jaGFydBIMGgp1ZHAtYWRkb25zChQKCGhlcml0YWdlEggaBlRpbGxlcgo2CilpbnN0YWxsLm9wZXJhdG9yLmlzdGlvLmlvL293bmluZy1yZXNvdXJjZRIJGgd1bmtub3duCiEKBWlzdGlvEhgaFmRhZ3N0ZXItaW5ncmVzc2dhdGV3YXkKGQoMaXN0aW8uaW8vcmV2EgkaB2RlZmF1bHQKMAobb3BlcmF0b3IuaXN0aW8uaW8vY29tcG9uZW50EhEaD0luZ3Jlc3NHYXRld2F5cwohChFwb2QtdGVtcGxhdGUtaGFzaBIMGgo1OThiYzU5OGQ5ChIKB3JlbGVhc2USBxoFaXN0aW8KOwofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIYGhZkYWdzdGVyLWluZ3Jlc3NnYXRld2F5Ci8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAoRCgNzaGESChoINjE0ZWE5MmMKIgoXc2lkZWNhci5pc3Rpby5pby9pbmplY3QSBxoFZmFsc2UKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCjEKBE5BTUUSKRonZGFnc3Rlci1pbmdyZXNzZ2F0ZXdheS01OThiYzU5OGQ5LXA3enc0ChsKCU5BTUVTUEFDRRIOGgxpc3Rpby1zeXN0ZW0KXwoFT1dORVISVhpUa3ViZXJuZXRlczovL2FwaXMvYXBwcy92MS9uYW1lc3BhY2VzL2lzdGlvLXN5c3RlbS9kZXBsb3ltZW50cy9kYWdzdGVyLWluZ3Jlc3NnYXRld2F5ChcKEVBMQVRGT1JNX01FVEFEQVRBEgIqAAopCg1XT1JLTE9BRF9OQU1FEhgaFmRhZ3N0ZXItaW5ncmVzc2dhdGV3YXk=’
‘x-envoy-peer-metadata-id’, ‘router~100.112.75.76~dagster-ingressgateway-598bc598d9-p7zw4.istio-system~istio-system.svc.cluster.local’
‘x-envoy-attempt-count’, ‘1’
‘x-b3-traceid’, ‘6884eec06668989ccb5ed70c16ef6f58’
‘x-b3-spanid’, ‘cb5ed70c16ef6f58’
‘x-b3-sampled’, ‘0’
‘x-envoy-original-path’, ‘/graphql’
2022-03-23T15:25:47.203134Z debug   envoy pool  queueing stream due to no available connections
2022-03-23T15:25:47.203139Z debug   envoy pool  trying to create new connection
2022-03-23T15:25:47.203144Z debug   envoy pool  creating a new connection
2022-03-23T15:25:47.203205Z debug   envoy client    [C8275] connecting
2022-03-23T15:25:47.203214Z debug   envoy connection    [C8275] connecting to 100.112.29.212:3000
2022-03-23T15:25:47.203322Z debug   envoy connection    [C8275] connection in progress
2022-03-23T15:25:47.203366Z debug   envoy http  [C8274][S2813519150144299458] request end stream
2022-03-23T15:25:47.204011Z debug   envoy connection    [C8275] connected
2022-03-23T15:25:47.205067Z debug   envoy client    [C8275] connected
2022-03-23T15:25:47.205080Z debug   envoy pool  [C8275] attaching to next stream
2022-03-23T15:25:47.205085Z debug   envoy pool  [C8275] creating stream
2022-03-23T15:25:47.205093Z debug   envoy router    [C8274][S2813519150144299458] pool ready
2022-03-23T15:25:47.205927Z debug   envoy connection    [C8275] remote close
2022-03-23T15:25:47.205937Z debug   envoy connection    [C8275] closing socket: 0
2022-03-23T15:25:47.205959Z debug   envoy connection    [C8275] SSL shutdown: rc=1
2022-03-23T15:25:47.205999Z debug   envoy client    [C8275] disconnect. resetting 1 pending requests
2022-03-23T15:25:47.206006Z debug   envoy client    [C8275] request reset
2022-03-23T15:25:47.206013Z debug   envoy router    [C8274][S2813519150144299458] upstream reset: reset reason: connection termination, transport failure reason:
2022-03-23T15:25:47.206043Z debug   envoy http  [C8274][S2813519150144299458] Sending local reply with details upstream_reset_before_response_started{connection termination}
2022-03-23T15:25:47.206079Z debug   envoy http  [C8274][S2813519150144299458] encoding headers via codec (end_stream=false):
‘:status’, ‘503’
‘content-length’, ‘95’
‘content-type’, ‘text/plain’
‘date’, ‘Wed, 23 Mar 2022 15:25:47 GMT’
‘server’, ‘istio-envoy’
2022-03-23T15:25:47.206202Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=12
2022-03-23T15:25:47.206213Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=6
2022-03-23T15:25:47.206217Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=10
2022-03-23T15:25:47.206221Z debug   envoy wasm  wasm log stats_outbound stats_outbound: [extensions/stats/plugin.cc:621]::report() metricKey cache hit , stat=14
2022-03-23T15:25:47.206232Z debug   envoy pool  [C8275] client disconnected, failure reason:
2022-03-23T15:25:47.206253Z debug   envoy pool  [C8275] destroying stream: 0 remaining

Where can we go to find more information about this error (logging, pods/locations, etc.)?

Answer 1

The error above came from an alternate (unsecured) service. The Istiod ads logs had the following:

    "pilot_conflict_inbound_listener": {
        "{{pod}}.{{namespace}}": {
            "proxy": "{{pod}}.{{namespace}}",
            "message": "Conflicting inbound listener:{{port}}. existing: {{insecured service}}.{{namespace}}.svc.cluster.local, incoming: {{actual service}}.{{namespace}}.svc.cluster.local"
        }
    }
}

Removing the service resolved the 503 response.