Istio-injected pod can't communicate with headless service in kubernetes

Question

For some reasons, I need an istio-injected namespace on my k8s environment, which will contain a number of pods. I also have pods outside this namespace, and a rabbitmq deployment using a headless service to be used as an interface for the rabbitmq replicas.

The fact is that the pods that are not istio-injected can communicate normally with the headless-service, but the istio-injected ones are not.

Istio-injected pods communicate through the istio-proxy sidecar, but I'm not 100% clear on why this blurs the headless service communication.

I've read in some places about the usage of a ServiceEntry, adding the headless service to the internal mesh that istio-proxy uses, but I haven't made it work so far. My file follows:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: rabbitmq-headless
  namespace: default
spec:
  hosts:
  - rabbitmq-headless.default.svc.cluster.local
  location: MESH_INTERNAL
  ports:
    - name: epmd
      protocol: TCP
      number: 4369
    - name: amqp
      protocol: TCP
      number: 5672
    - name: dist
      protocol: TCP
      number: 25672
    - name: http-stats
      protocol: TCP
      number: 15672
  resolution: DNS

With or without the Service entry, the application shows: System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (111): Connection refused 10.72.2.3:5672

10.72.2.3:5672 is the related endpoint in my headless-service, pointing to the ip of one of the rabbitmq replicas of the deployment: rabbitmq-headless-service-endpoints

[UPDATE] I tried various strategies, with a variety of istio resources, and really serviceEntry seems to be the only one that allows me to manipulate the communication permissions on the istio-proxy. I installed istioctl and used istioctl proxy-config cluster -n <namespace> <pod-name>, from where i can see the outbound rules i've added via serviceEntry. I have manipulated the host names, ports, protocols, resolution types, and have seem the changes being applied to the rules. Unfortunately, even adding the rules, the connections still cannot be made.

[UPDATE-2] I was able to apply all configs i wanted via ServiceEntry or Sidecar istio resources. I was also able to see the detailed logs from istio accusing the correct auth-method and dns resolution, as well as the connection rejection by the rabbitmq server, and Im troubleshooting it now.

SOLUTION

• After finding various references, i investigated the most probable problems:

  • Wrong communication protocol: Rabbitmq default 5672 works with simple TCP, not http. The fact is that when the connection is in plain text, the TCP is supported by default, so that's not the problem.
  • Mtls forced but not supported by the rabbitmq server (on default 5672): In my istio version, the service to service communication is by default mtls PERMISSIVE (can be explicit in a PeerAuthentication resource file), so, when istio-proxy communicates with a non-injected pod, it uses plain text, so enforced mtls is not our problem.
  • Impossible DNS resolution: The istio injected mesh could not be seeing outside services (outside the mesh), or even, the mechanisms used to translate to final pod ip could not be working properly. By using istioctl pc log -n <namespace> <pod-name> --level upstream:debug,connection:debug,envoy_bug:debug,config:debug,admin:debug,alternate_protocols_cache:debug,dns:debug i could see proper logging accusing address resolution and connection attempts. This way, i was able to discard problems in these steps.

• So, after all, i tested a rabbitmq interface focused on resilience (forces the connection repeatedly), and it worked. Why? The istio-proxy sidecar is deployed at the same time that the application container, so, it can try to connect, before the istio-proxy is ready to properly work out that connection. The resilient interface sees no problem in it, once it retries until it reaches the server.

• A good way to diagnose this is to put a sleep command in your application dockerfile, before the startup command, so the application only reaches the fully booted proxy sidecar.

• I hope it helps whoever may be experiencing similar problems, and remind everyone, again, that a k8s with istio environment has to be used always with the idea of ephemeral resources, as my problem was an interface that relied on a stable, always reachable, connection.