Prometheus invalid metric type "status=\"available\"} gauge" + 401

Question

Trying to monitor with Prometheus, but cannot get /_admin/metrics/v2/ to work with jwt (I've confirmed the token matches web, tested with postman). So I've enabled the arango-prometheus-exporter service, and can read metrics from the mount point /metrics.

However, Prometheus is now reporting invalid metric type "status=\"available\"} gauge" in the targets list page, and the endpoint is shown as down in red.

I am unsure of 3 things:

1. Why do I get a 401 not authorized to execute this request when using a valid jwt bearer token,
2. Why am i getting invalid metric type "status=\"available\"} gauge" for the prometheus target endpoint using the foxx-service mounted at /metrics,
3. Should i even be using the foxx-service, or the internal admin metrics url?

I am using the latest as of a couple days ago, ArangoDB v3.10.1, Prometheus v2.41.0-rc.0 (and Grafana v9.3.2 (21c1d14e91) but i haven't gotten that far with ArangoDB yet). All on Ubuntu 20.04.4 LTS.

I am currently successfully monitoring Prometheus, Node exporter and RabbitMQ.

UPDATE: I should be using the ArangoDB metrics rather than Foxx. More info on issues using built-in metrics:

I have generated a secret from arangodb on command line using arangodb create jwt-secret, then used arangodb auth header --auth.jwt-secret secret.jwt to generate the auth header string. I copy the token to /etc/prometheus/prometheus.token and update prometheus.yml to:

  - job_name: "arangodb"
    scrape_interval: 5s
    metrics_path: /_admin/metrics/v2
    bearer_token_file: /etc/prometheus/prometheus.token
    static_configs:
      - targets: ["localhost:8529"]

Then i restart prometheus to pick up the changes, and in the Targets tab i see

I tested by running as sudo to generate the secret from ArangoDB, then sudo to generate the Bearer Token, but still the same result.

However, if I copy the token from the jwt key from the Application tab in devtools from the ArangoDB site and paste it into Postman Authorisation section under Bearer Token and make a GET request to http://172.31.245.140:8529/_admin/metrics/v2/ it does work.

So I copied the same token from website login to /etc/prometheus/prometheus.token, restart Prometheus and now its working. But, it will only last until the web token times out. <- And confirmed, jwt token has timed out and Prometheus is showing "server returned HTTP status 401 Unauthorized" again.

Am I missing something?

UPDATE 2 - I can connect using basic auth in the prometheus config.

So this works:

  - job_name: "arangodb"
    scrape_interval: 5s
    metrics_path: /_admin/metrics/v2
    #bearer_token_file: /etc/prometheus/prometheus.token
    basic_auth:
      username: root
      password: the_password
    static_configs:
      - targets: ["localhost:8529"]

I didn't try password_file instead, but that would at the least remove password from the config file. I'm ok with this for now since it will be locked down, but i would rather use token auth.