I am trying to build a simple application that sends traces to OpenTelemetry Collector, which exports the traces to Jaeger Backend.
But while I spin up the collector and Jaeger Backend, I get the following message,
info jaegerexporter/exporter.go:186 State of the connection with the Jaeger Collector backend {"kind": "exporter", "name": "jaeger", "state": "TRANSIENT_FAILURE"}
When I run the go application, I see no traces on the Jaeger UI. Also, no logs from the collector the shell.
main.go
package main
import (
"context"
"fmt"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
)
func initialize() {
traceExp, err := otlptracehttp.New(
context.TODO(),
otlptracehttp.WithEndpoint("0.0.0.0:55680"),
otlptracehttp.WithInsecure(),
)
if err != nil {
fmt.Println(err)
}
bsp := sdktrace.NewBatchSpanProcessor(traceExp)
tracerProvider := sdktrace.NewTracerProvider(
sdktrace.WithSpanProcessor(bsp),
)
otel.SetTracerProvider(tracerProvider)
}
func main() {
initialize()
tracer := otel.Tracer("demo-client-tracer")
ctx, span := tracer.Start(context.TODO(), "span-name")
defer span.End()
time.Sleep(time.Second)
fmt.Println(ctx)
}
Following are the collector config and docker-compose file.
otel-collector-config
receivers:
otlp:
protocols:
http:
processors:
batch:
exporters:
jaeger:
endpoint: "http://jaeger-all-in-one:14250"
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [jaeger]
docker-compose.yaml
version: "2"
services:
# Jaeger
jaeger-all-in-one:
image: jaegertracing/all-in-one:latest
ports:
- "16686:16686"
- "14268"
- "14250:14250"
# Collector
otel-collector:
image: otel/opentelemetry-collector:latest
command: ["--config=/etc/otel-collector-config.yaml"]
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
ports:
- "4317"
- "55680:55680"
depends_on:
- jaeger-all-in-one
Additional Logs while running docker-compose up,
Starting open-telemetry-collector-2_jaeger-all-in-one_1 ... done
Starting open-telemetry-collector-2_otel-collector_1 ... done
Attaching to open-telemetry-collector-2_jaeger-all-in-one_1, open-telemetry-collector-2_otel-collector_1
jaeger-all-in-one_1 | 2021/09/02 09:26:58 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0155272,"caller":"flags/service.go:117","msg":"Mounting metrics handler on admin server","route":"/metrics"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.015579,"caller":"flags/service.go:123","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.016236,"caller":"flags/admin.go:106","msg":"Mounting health check on admin server","route":"/"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0163133,"caller":"flags/admin.go:117","msg":"Starting admin HTTP server","http-addr":":14269"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0163486,"caller":"flags/admin.go:98","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.017912,"caller":"memory/factory.go:61","msg":"Memory storage initialized","configuration":{"MaxTraces":0}}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.018202,"caller":"static/strategy_store.go:138","msg":"Loading sampling strategies","filename":"/etc/jaeger/sampling_strategies.json"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0273001,"caller":"server/grpc.go:82","msg":"Starting jaeger-collector gRPC server","grpc.host-port":":14250"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0273921,"caller":"server/http.go:48","msg":"Starting jaeger-collector HTTP server","http host-port":":14268"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0276191,"caller":"server/zipkin.go:49","msg":"Not listening for Zipkin HTTP traffic, port not configured"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0276558,"caller":"grpc/builder.go:70","msg":"Agent requested insecure grpc connection to collector(s)"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0276873,"caller":"channelz/logging.go:50","msg":"[core]parsed scheme: \"\"","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0277174,"caller":"channelz/logging.go:50","msg":"[core]scheme \"\" not registered, fallback to default scheme","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0277457,"caller":"channelz/logging.go:50","msg":"[core]ccResolverWrapper: sending update to cc: {[{:14250 <nil> 0 <nil>}] <nil> <nil>}","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0277772,"caller":"channelz/logging.go:50","msg":"[core]ClientConn switching balancer to \"round_robin\"","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0277963,"caller":"channelz/logging.go:50","msg":"[core]Channel switches to new LB policy \"round_robin\"","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0278597,"caller":"grpclog/component.go:55","msg":"[balancer]base.baseBalancer: got new ClientConn state: {{[{:14250 <nil> 0 <nil>}] <nil> <nil>} <nil>}","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0279217,"caller":"channelz/logging.go:50","msg":"[core]Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.028044,"caller":"channelz/logging.go:50","msg":"[core]Subchannel picks a new address \":14250\" to connect","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0284538,"caller":"grpclog/component.go:71","msg":"[balancer]base.baseBalancer: handle SubConn state change: 0xc000688840, CONNECTING","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.028513,"caller":"channelz/logging.go:50","msg":"[core]Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0280442,"caller":"grpc/builder.go:109","msg":"Checking connection to collector"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.028587,"caller":"grpc/builder.go:120","msg":"Agent collector connection state change","dialTarget":":14250","status":"CONNECTING"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0294988,"caller":"channelz/logging.go:50","msg":"[core]Subchannel Connectivity change to READY","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.029561,"caller":"grpclog/component.go:71","msg":"[balancer]base.baseBalancer: handle SubConn state change: 0xc000688840, READY","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0296533,"caller":"grpclog/component.go:71","msg":"[roundrobin]roundrobinPicker: newPicker called with info: {map[0xc000688840:{{:14250 <nil> 0 <nil>}}]}","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0297205,"caller":"channelz/logging.go:50","msg":"[core]Channel Connectivity change to READY","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0297425,"caller":"grpc/builder.go:120","msg":"Agent collector connection state change","dialTarget":":14250","status":"READY"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0298278,"caller":"./main.go:233","msg":"Starting agent"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0298927,"caller":"querysvc/query_service.go:137","msg":"Archive storage not created","reason":"archive storage not supported"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0299237,"caller":"app/flags.go:124","msg":"Archive storage not initialized"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0300004,"caller":"app/agent.go:69","msg":"Starting jaeger-agent HTTP server","http-port":5778}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0303733,"caller":"channelz/logging.go:50","msg":"[core]parsed scheme: \"\"","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0304158,"caller":"channelz/logging.go:50","msg":"[core]scheme \"\" not registered, fallback to default scheme","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0304341,"caller":"channelz/logging.go:50","msg":"[core]ccResolverWrapper: sending update to cc: {[{:16685 <nil> 0 <nil>}] <nil> <nil>}","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0304427,"caller":"channelz/logging.go:50","msg":"[core]ClientConn switching balancer to \"pick_first\"","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0304537,"caller":"channelz/logging.go:50","msg":"[core]Channel switches to new LB policy \"pick_first\"","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0305033,"caller":"channelz/logging.go:50","msg":"[core]Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0305545,"caller":"channelz/logging.go:50","msg":"[core]Subchannel picks a new address \":16685\" to connect","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"warn","ts":1630574818.0307937,"caller":"channelz/logging.go:75","msg":"[core]grpc: addrConn.createTransport failed to connect to {:16685 localhost:16685 <nil> 0 <nil>}. Err: connection error: desc = \"transport: Error while dialing dial tcp :16685: connect: connection refused\". Reconnecting...","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.030827,"caller":"channelz/logging.go:50","msg":"[core]Subchannel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0308597,"caller":"grpclog/component.go:71","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc00061fd40, {CONNECTING <nil>}","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0308924,"caller":"channelz/logging.go:50","msg":"[core]Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0309658,"caller":"grpclog/component.go:71","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc00061fd40, {TRANSIENT_FAILURE connection error: desc = \"transport: Error while dialing dial tcp :16685: connect: connection refused\"}","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0309868,"caller":"channelz/logging.go:50","msg":"[core]Channel Connectivity change to TRANSIENT_FAILURE","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0314078,"caller":"app/static_handler.go:181","msg":"UI config path not provided, config file will not be watched"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0315406,"caller":"app/server.go:197","msg":"Query server started"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0315752,"caller":"healthcheck/handler.go:129","msg":"Health Check state change","status":"ready"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0315914,"caller":"app/server.go:276","msg":"Starting GRPC server","port":16685,"addr":":16685"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574818.0316222,"caller":"app/server.go:257","msg":"Starting HTTP server","port":16686,"addr":":16686"}
jaeger-all-in-one_1 | {"level":"info","ts":1630574819.031331,"caller":"channelz/logging.go:50","msg":"[core]Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574819.0314019,"caller":"channelz/logging.go:50","msg":"[core]Subchannel picks a new address \":16685\" to connect","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574819.0315094,"caller":"grpclog/component.go:71","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc00061fd40, {CONNECTING <nil>}","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574819.0315537,"caller":"channelz/logging.go:50","msg":"[core]Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574819.0323153,"caller":"channelz/logging.go:50","msg":"[core]Subchannel Connectivity change to READY","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574819.0325227,"caller":"grpclog/component.go:71","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc00061fd40, {READY <nil>}","system":"grpc","grpc_log":true}
jaeger-all-in-one_1 | {"level":"info","ts":1630574819.0325499,"caller":"channelz/logging.go:50","msg":"[core]Channel Connectivity change to READY","system":"grpc","grpc_log":true}
otel-collector_1 | 2021-09-02T09:26:59.628Z info service/collector.go:303 Starting otelcol... {"Version": "v0.33.0", "NumCPU": 8}
otel-collector_1 | 2021-09-02T09:26:59.628Z info service/collector.go:242 Loading configuration...
otel-collector_1 | 2021-09-02T09:26:59.630Z info service/collector.go:258 Applying configuration...
otel-collector_1 | 2021-09-02T09:26:59.630Z info builder/exporters_builder.go:264 Exporter was built. {"kind": "exporter", "name": "jaeger"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info builder/pipelines_builder.go:214 Pipeline was built. {"pipeline_name": "traces", "pipeline_datatype": "traces"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info builder/receivers_builder.go:227 Receiver was built. {"kind": "receiver", "name": "otlp", "datatype": "traces"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info service/service.go:143 Starting extensions...
otel-collector_1 | 2021-09-02T09:26:59.630Z info service/service.go:188 Starting exporters...
otel-collector_1 | 2021-09-02T09:26:59.630Z info builder/exporters_builder.go:93 Exporter is starting... {"kind": "exporter", "name": "jaeger"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info jaegerexporter/exporter.go:186 State of the connection with the Jaeger Collector backend {"kind": "exporter", "name": "jaeger", "state": "CONNECTING"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info builder/exporters_builder.go:98 Exporter started. {"kind": "exporter", "name": "jaeger"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info service/service.go:193 Starting processors...
otel-collector_1 | 2021-09-02T09:26:59.630Z info builder/pipelines_builder.go:52 Pipeline is starting... {"pipeline_name": "traces", "pipeline_datatype": "traces"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info builder/pipelines_builder.go:63 Pipeline is started. {"pipeline_name": "traces", "pipeline_datatype": "traces"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info service/service.go:198 Starting receivers...
otel-collector_1 | 2021-09-02T09:26:59.630Z info builder/receivers_builder.go:71 Receiver is starting... {"kind": "receiver", "name": "otlp"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info otlpreceiver/otlp.go:93 Starting HTTP server on endpoint 0.0.0.0:4318 {"kind": "receiver", "name": "otlp"}
otel-collector_1 | 2021-09-02T09:26:59.630Z info otlpreceiver/otlp.go:159 Setting up a second HTTP listener on legacy endpoint 0.0.0.0:55681 {"kind": "receiver", "name": "otlp"}
otel-collector_1 | 2021-09-02T09:26:59.631Z info otlpreceiver/otlp.go:93 Starting HTTP server on endpoint 0.0.0.0:55681 {"kind": "receiver", "name": "otlp"}
otel-collector_1 | 2021-09-02T09:26:59.631Z info builder/receivers_builder.go:76 Receiver started. {"kind": "receiver", "name": "otlp"}
otel-collector_1 | 2021-09-02T09:26:59.631Z info service/collector.go:206 Setting up own telemetry...
otel-collector_1 | 2021-09-02T09:26:59.631Z info service/telemetry.go:99 Serving Prometheus metrics {"address": ":8888", "level": 0, "service.instance.id": "0fe56a33-e40e-4251-9a82-100fa600c4a0"}
otel-collector_1 | 2021-09-02T09:26:59.631Z info service/collector.go:218 Everything is ready. Begin running and processing data.
otel-collector_1 | 2021-09-02T09:27:00.631Z info jaegerexporter/exporter.go:186 State of the connection with the Jaeger Collector backend {"kind": "exporter", "name": "jaeger", "state": "TRANSIENT_FAILURE"}
Thanks!
Updating otel-collector-config.yaml to the following endpoint should work: