I'm using ELK of version 7.17. Running different Celery tasks and logging "Task started", "Task finished" events to the Elasticsearch via Logstash. I'm adding a trace_id field as log_extra, so, I can recognize pairs related to single task. I want to get an average time spent per task_type and how did it change in specified period?
I read it's possible with Timelion, but everything I tried fails... is it possible at all?
UPD 1:
Example documents:
{
"@timestamp": "2023-05-23T06:01:01.111Z",
"message": "Task started",
"trace_id": "1",
"task_type": "A"
}
{
"@timestamp": "2023-05-23T06:02:02.222Z",
"message": "Task started",
"trace_id": "2",
"task_type": "A"
}
{
"@timestamp": "2023-05-23T06:03:03.333Z",
"message": "Task finished",
"trace_id": "2",
"task_type": "A"
}
{
"@timestamp": "2023-05-23T06:04:04.444Z",
"message": "Task started",
"trace_id": "3",
"task_type": "B"
}
{
"@timestamp": "2023-05-23T06:05:05.555Z",
"message": "Task finished",
"trace_id": "1",
"task_type": "A"
}
{
"@timestamp": "2023-05-23T06:06:06.666Z",
"message": "Task finished",
"trace_id": "3",
"task_type": "B"
}
In this example we have 3 Celery tasks of 2 types - "A" and "B". First task "A" with trace_id=1 start and write log message "Task started", then another task of type "A" (trace_id=2) start and finish until the end of the first task. Then a third task of type "B"(trace_id=3) start and will be finished only after the first task will finish.