im not able to collect traces using OpenTelemetry-exporters.
The Setup:
- Jaeger Helm-Chart running on kubernetes in Azure
- Apache Flink Job (java) using OpenTelemetry javaagent like described here
env variables are seated like this:
OTEL_EXPORTER_JAEGER_ENDPOINT=http://jaeger-collector.jaeger:14250
OTEL_TRACES_EXPORTER=jaeger
OTEL_LOGS_EXPORTER=none
OTEL_METRICS_EXPORTER=none
The Jaeger-collector is reachable in-cluster over the configured URL and also the port is exposed and running.
Where spans should be collected I receive the following error message:
2023-10-24 11:22:27.887
io.jaegertracing.internal.exceptions.SenderException: Failed to flush spans.
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:116) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.internal.reporters.RemoteReporter$FlushCommand.execute(RemoteReporter.java:158) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.internal.reporters.RemoteReporter$QueueProcessor.run(RemoteReporter.java:179) [pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at java.lang.Thread.run(Unknown Source) [?:?]
2023-10-24 11:22:27.887
Caused by: io.jaegertracing.internal.exceptions.SenderException: Could not send 7 spans
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:86) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
... 3 more
2023-10-24 11:22:27.887
Caused by: org.apache.thrift.transport.TTransportException: Cannot flush closed transport
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:151) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:84) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
... 3 more
2023-10-24 11:22:27.887
Caused by: java.net.PortUnreachableException: ICMP Port Unreachable
2023-10-24 11:22:27.887
at java.net.PlainDatagramSocketImpl.send0(Native Method) ~[?:?]
2023-10-24 11:22:27.887
at java.net.AbstractPlainDatagramSocketImpl.send(Unknown Source) ~[?:?]
2023-10-24 11:22:27.887
at java.net.DatagramSocket.send(Unknown Source) ~[?:?]
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.reporters.protocols.ThriftUdpTransport.flush(ThriftUdpTransport.java:149) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at org.apache.thrift.TServiceClient.sendBaseOneway(TServiceClient.java:66) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.agent.thrift.Agent$Client.send_emitBatch(Agent.java:70) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.agent.thrift.Agent$Client.emitBatch(Agent.java:63) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.senders.UdpSender.send(UdpSender.java:84) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
at io.jaegertracing.thrift.internal.senders.ThriftSender.flush(ThriftSender.java:114) ~[pcmasfd-zip-group-state-aggregator-1.1.235-users-pcg-num-produced-parts-flink-way.4.jar:?]
2023-10-24 11:22:27.887
... 3 more
I have also tried to configure the OpenTelemetry exporter (instead of jaeger-exporter) with the otel-grpc ports (4317) but receiving the same.
Does anyone have an idea what's wrong here?