I'm looking into the possibilities of monitoring hadoop cluster with ELK/EFK stack. I have searched over the public domains but couldn't find anything relevant.

Any help in this regard will be highly appreciated

1 Answers

cricket_007 On

It's not clear what you're trying to monitor.

Everything in Hadoop is mostly a Java process, so adding some JMX exporters like Prometheus or Jolokia would expose metrics over REST, and from there you would have to periodically poll those into Elasticsearch.

To enable JMX, you'd have to edit the hadoop-env.sh scripts, I believe, for YARN and HDFS, to control any JVM options. Hive, Spark, Hbase, etc all have similar scripts

General example here on Jolokia https://www.elastic.co/blog/monitoring-java-applications-with-metricbeat-and-jolokia

Other than that, Filebeat and Metricbeat operate the same as any other system

If you used Cloudera Manager or Ambari to control your cluster, then monitoring would be provided for you from those tools