I'm trying to export spark (2.4.0) custom metrics in prometheus format. I found this guide https://argus-sec.com/monitoring-spark-prometheus/ Running on my local pc (windows) I am able to see my metrics in visualVM and in spark's json http://localhost:4040/metrics/json/
My issue is that the JMX exporter (https://github.com/prometheus/jmx_exporter/blob/master/README.md) does not show any signs that it's active at all. I added the following to spark conf :
val conf = new SparkConf()
val metrics = Map("spark.metrics.conf" -> "<pathOnMyPC>\metrics.properties")
val prometheus = Map("spark.driver.extraJavaOptions" -> "spark.driver.extraJavaOptions=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8090 -Dcom.sun.management.jmxremote.rmi.port=8091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost -javaagent:<pathOnMyPC>jmx_prometheus_javaagent-0.13.0.jar=8095:<pathOnMyPC>\prometheus-config.yaml")
conf.setAll(metrics)
conf.setAll(prometheus)
And my prometheus-config.yaml file has no rules to present everything just to see that the data is extracted.
---
lowercaseOutputName: true
attrNameSnakeCase: true
I do not see any logs from the JMX exporter and http://localhost:8095/metrics can’t be reached. Please advise :)