Disabling proxy protocol for the in-cluster communication in Istio

Question

I have istio setup behind a LoadBalancer with proxy protocol, so I have to setup following EnvoyProxy:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: LISTENER
      patch:
        operation: MERGE
        value:
          listener_filters:
            - name: envoy.filters.listener.proxy_protocol
            - name: envoy.filters.listener.tls_inspector
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingressgateway-settings
  namespace: istio-system
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
      patch:
        operation: MERGE
        value:
          name: envoy.http_connection_manager
          typed_config:
            "@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
            skip_xff_append: false
            use_remote_address: true
            xff_num_trusted_hops: 1

Everything works, but the problem is that affects in-cluster communication too. When communicating inside cluster, I don't want to use a proxy protocol. It affects my use case where I want to setup VirtualService which will then route requests to the relevant services, which are also istio entities (specifically, Istio VirtualService/ knative services):

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: "app-virtual-service"
spec:
  gateways:
    - knative-serving/knative-local-gateway
    - "app-gateway"
  hosts:
    - hub.example.com
  http:
    # GraphQL API
    - match:
        - uri:
            prefix: "/graphql"
      rewrite:
        authority: app-api.app.svc.cluster.local
      route:
        - destination:
            host: app-api.app.svc.cluster.local
          weight: 100
    # Hall service
    - match:
        - uri:
            prefix: "/hall"
      rewrite:
        authority: app-hall.app.svc.cluster.local
      route:
        - destination:
            host: app-hall.app.svc.cluster.local
          weight: 100
    # Manager frontend
    - match:
        - uri:
            exact: /
        {{- range .Values.manager.paths }}
        - uri:
            prefix: {{ . | quote }}
        {{- end }}
      rewrite:
        authority: app-manager.app.svc.cluster.local
      route:
        - destination:
            host: app-manager.app.svc.cluster.local
          weight: 100

Whenever I try to query http://hub.example.com/graphql, for example, it tries to query app-api.app.svc.cluster.local (which is Istio VirtualService) without proxy protocol (when it is expected due to the rule, or at least that's how I understand it). So that fails with this error:

upstream connect error or disconnect/reset before headers. reset reason: connection termination

If I disable proxy rules it all works fine and if I try to access knative service directly from the LoadBalancer it also works.

What doesn't work is whenever some VirtualService tries to forward request to another one. So my scheme looks like this

External LoadBalancer -> Istio Ingress LoadBalancer - Gateway - VirtualService (accepts proxy protocol payload) -> End-service VirtualService (also accepts proxy protocol payload, but the service before is sending raw query)

Is there any way to avoid using proxy protocol inside cluster communication? Possible solutions I've thought about is creating multiple gateways, but I'm not sure how should I go about that and use that inside a filter.

Answer 1

In this official Istio documentation they have explained how to bypass proxy for a specific IP range, global.proxy.includeIPRanges and global.proxy.excludeIPRanges options will help you in bypassing the envoy proxy. In the documentation they have explained how to bypass proxy for external services and you need to do the reverse of it. Check the internal IP range of the cluster and use global.proxy.excludeIPRanges option and pass the internal IP range to this command. As mentioned in the below command

kubectl describe pod kube-apiserver -n kube-system | grep 'service-cluster-ip-range' 
#command for getting the IP range of the cluster

istioctl manifest apply <the flags you used to install Istio> --set values.global.proxy.excludeIPRanges="IP range"
#command for bypassing the internal IP range from envoy proxy