syslog data valuable for machine learning?

Question

Does anyone have experience with syslog data and machine learning or deep learning? Is syslog data rich enough to use in machine learning algorithms and gain insights? If so, what insights can be gained for it? Are there tools available currently to make sense of vast volumes of syslog data?

Answer 1

Splunk can do this relatively easily, however the data scrubbing part can take quite awhile. You will need to tie Splunk into a lot for this, because you're talking about operational intelligence machine learning, which is gobs of data and systems.

To answer your question, yes. I've used syslog data to track anomalous failures, and leveraged machine learning to predict (and reduce) number of outages due to ticket entry. to say this another way, we used machine learning to predict how many people would panic during a scheduled (and communicated) outage, so we could make accurate our reporting metrics.

I can say it's valuable, but if you're talking about predicting database connectivity issue algorithms, or application issues, it's at best a factor. syslog isn't dense enough to tell you what's happening in every application running on the machine (unless you classify any log you send to /var/log 'syslog') application logs need to be taken into consideration when performing machine learning algorithms, simply because there is no better source of activity for that application. The trick is determining the dependencies of that application or service.

Splunk is a great tool, and pretty easy to use in comparison to others for making sense of gobs of data. I use it to scub all syslog files constantly for connection issues, and it's super easy after the data is in.