Reading Graphite events from Icinga2

Question

In our lab we are currently using two systems to remotely monitor sensitive quantities etc. :

• Icinga2, as a monitoring system that reads data from a logging DAQ called Watchdog and creates alarms when measured values exceed user-defined thresholds;

• Graphite to plot data from Carbon, where it had been received via network or otherwise (saved on disk) and stored, onto a webpage so as to be accessible remotely.

So far we have only been able to have Icinga2 write whatever it received on Graphite, so that we can see it.

Is it possible to have Icinga2 read from Graphite?
There are other programs that are writing things to the Graphite directly, and that cannot go through the DAQ Watchdog to be read from Icinga (lack of ports).
Ideally we would like to create alarms (that Icinga2 can do) on whatever reaches the Graphite.

Answer 1

I think I found a solution in the end: as a part of the Graphite package installation there is the Whisper database. This inclues a number of useful scripts, like whisper-fetch.py which allows to exctract data from the .wsp file (where the plotted graphite-web is stored) to any format we want.

One just needs to check that the whole Whisper package has been installed, which can be done just by checking whether import whisper on a python terminal does not return an error. If not fully installed, checkout the whisper on GitHub.

Answer 2

There are several check_graphite plugin implementations around which allow you to query the API provided by Graphite web. Unfortunately I don't have an recommendation here, try Google hits which fits your needs at best.

All of them connect to the Graphite web API:

http://graphite.readthedocs.io/en/latest/render_api.html

Graphite is really strong on its own already, you can let it aggregate data points and reduce the logic in your plugin to just evaluate given thresholds and return output and state.

If you are using Grafana, you can also look into their alert and notification conditions and let e.g. the web hook pass back check results to Icinga 2's REST API.

http://docs.grafana.org/alerting/rules/ http://docs.grafana.org/alerting/notifications/