Want architecture for storage and tracking of application metrics

Question

Like many modern applications I have several moving pieces as part of my current application:

• Web service
• Various queues
• Various worker processes
• etc.

To properly manage my application I want to track various arbitrary application-related metrics such as:

• Average queue length over some period of time
• Average queue processing time and/or max processing time
• Number of items processed per unit of time, or number of items of type X processed per unit of time, e.g. how many in last minute, hour, day
• and so on

I'm having trouble coming up with a logical model for this and then an actual implementation. Some of the things I'm struggling with:

• How are these calculations made? By the same processes that are doing the things I am measuring? By a separate process?
• When are these calculations made? Certainly the calculations shouldn't be synchronous with the application flow for example.
• How am I storing the results of these calculations? Is there a database schema that lends itself to storing metrics like this?

Part of me feels like this is a solved problem and there is an architecture or pattern I should be adopting or reusing.

I ask this question purposely without mentioning the specific technologies my application is using because my gut tells me that isn't important to the pattern.

Thoughts?

Answer 1

Here are some pointers for each questions

How are these calculations made? By the same processes that are doing the things I am measuring? By a separate process?

Definitely not by the same process. The reason is, if you bind those calculations to any process that is not entirely serving this only purpose, you'll end with some scattered logic all over your services, and that will quickly become unmaintainable. Have a centralized place where all the calculation are performed. Have each piece of your achitecture sending their payload via some agnostic transfer, like REST (or something faster if you have like hundreds per second, like message queue as you mentioned).

When are these calculations made? Certainly the calculations shouldn't be synchronous with the application flow for example.

This depends on your use case. If you don't need to perform all calculations in real time, you can have a static component that receives all the incoming dataflow from your other actors, then stores them temporarily (more on that later), and another component that goes through all the newly acquired data (or all of it) to perform calculations. The latter can be scheduled by a library like Celery, or use standard cron jobs.

How am I storing the results of these calculations? Is there a database schema that lends itself to storing metrics like this?

Standard SQL could be used for that, almost any implementation of it. Now if you have metrics that are mostly time-stamped or time series data, you might take a look at Time Series Databases (TSDB).