Can we trace external API calls with Istio behind proxy via Kiali?

Question

We have a Nodejs based microservices running in our on-prem kubernetes v1.19 with Istio v1.8.0. What I would like to achieve is trace or display the external API calls in Kiali where we have Jaeger clients for each microservices and able to trace internal traffics.

But so far I could not able to trace any external API calls hits from any microservices.The only thing that I can see the traffic for proxy in Kiali's graph overview.

We have a cooperate proxy, and each container have env proxies set for both http_proxy, https_proxy.Any external service accessible via a cooperate proxy thus traffics should go through the our cooperate proxy first. We have a secured gateway with TLS and we do not have egressgateway where only have istio-ingressgateway.

So is there anyway to trace external traffics likewise the internal traffics inside cluster?If yes what might be the missing thing?

   $ kubectl get pods -n dev
    NAME                                     READY   STATUS    RESTARTS   AGE
    api-dev-74896ff4f9-slxt5                 3/3     Running   0          7h1m
    auth-dev-98f77d487-qt5zd                 3/3     Running   0          3d5h
    backend-dev-bb7765464-b7bpr              2/2     Running   0          7d3h
    mp-dev-86d6b8b978-slqp7                  3/3     Running   0          5d9h
    ui-dev-d5667946b-sdvlc                   2/2     Running   0          5d4h

Here are the ServiceEntries and VirtualServices that I created where I would like to use the retry feature as well the calls for proxy and externalAPI

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: company-proxy
  namespace: dev
spec:
  hosts:
  - foo-proxy.net
  ports:
  - number: PORT
    name: tcp
    protocol: TCP
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: proxy
  namespace: dev
spec:
  hosts:
    - "foo-proxy.net"
  http:
    - name: "company-proxy"
      match:
        - uri:
            prefix: "/"
      route:
        - destination:
            host: "foo-proxy.com"
      timeout: 90s
      retries:
        retryOn: "5xx"
        attempts: 3
        perTryTimeout: 30s
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: foo-example.com
  namespace: dev
spec:
  hosts:
    - "foo-example.com"
  ports:
    - number: 80
      name: http
      protocol: HTTP
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: foo-example.com
  namespace: dev
spec:
  hosts:
    - "foo-example.com"
  http:
    - name: "developer-api"
      match:
        - uri:
            prefix: "/"
      route:
        - destination:
            host: "foo-example.com"
      timeout: 90s
      retries:
        retryOn: "5xx"
        attempts: 3
        perTryTimeout: 30s

Answer 1

I am not sure why Istio doesn't automatically trace your calls to external APIs. Perhaps it requires an egress gateway to be used, I'm not sure. Note also that Istio creates traces for http(s) traffic, not TCP.

However, this is something you can still do programmatically. You can use any of the Jaeger client libraries to augment"the traces already created by Envoy by appending your own spans.

To do so, you need first to extract the trace context from the HTTP headers of the incoming request (assuming that your external API calls are consecutive to an incoming request), and then create a new span as child of that previous span context. A good idea would be to use OpenTracing semantic conventions when you tag your new span. Tools like Kiali will be able to leverage some information if it follows this convention.

I've found this blog post that explains how to do it with the nodejs jaeger client: https://rhonabwy.com/2019/01/06/adding-tracing-with-jaeger-to-an-express-application/