Observing average wait time for goroutines

Question

I'm running a server application serving a high number of QPS and for each query I perform some computation which is heavily CPU bounded.

I took a trace and put a screenshot below. The x-axis is the time and the y axis is the number of goroutines. In dark green, we can see the number of goroutine in Running state, and in light blue, we can see the number of goroutines in Runnable state.

From this, it's hard to know whether it's the same goroutines being queued or there is some context switch happening.
I'd like to know if it's possible to get the average wait time for goroutines (time they wait before they get to be scheduled).

Answer 1

Go exposes some runtime metrics there: https://pkg.go.dev/runtime/metrics

Among the metrics exposed is this one:

/sched/latencies:seconds
    Distribution of the time goroutines have spent in the scheduler
    in a runnable state before actually running.