Promtail pipeline labels not displayed in Grafana when using JSON parsing stage

281 views Asked by At

I'm trying to extract values from JSON formatted log lines using Promtail's pipeline to create labels for visualization in Grafana. However, upon querying in Grafana, the expected labels are not displayed. I used the explananation from the grafana docs: https://grafana.com/docs/loki/latest/send-data/promtail/stages/json/

JSON Log Format:

{"log":"log message\n","grpc.stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}

Promtail Configuration:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: https://sg-mrslog-01.schmidtgruppe.org:3100
    tls_config:
      insecure_skip_verify: true

scrape_configs:
  - job_name: test_1
    static_configs:
      - targets:
          - localhost
        labels:
          job: test_1
          system: mrs2-14
          __path__: /var/lib/mrs/data/log/test1.json
    pipeline_stages:
      - json:
          expressions:
            output: log
            stream: '"grpc.stream"'
            timestamp: time

Output in Grafana grafana query

Expected Behavior: When I query and click on a log entry in Grafana, I expect to see the labels:

output: "log message\n" stream: "stderr" timestamp: "2019-04-30T02:12:41.8443515Z" Actual Behavior: Upon selecting a log line in Grafana, the mentioned labels and their associated values are not displayed.

What I Tried:

Ensured that Loki is correctly set up as a data source in Grafana. Checked the Promtail setup to make sure it's correctly collecting and sending logs to Loki. Tried different parsing stages in the Promtail configuration, but still facing the same issue.
I also Used the json format in the grafana query: json formatter

Expected Outcome: I want to correctly visualize the extracted labels and their values in Grafana when selecting a log entry.

Is there any step I might be missing or something I might have configured wrongly?

Edit: I send the following json file via promtail:

{
  "Date": "2023-10-23T11:43:57.0096488+02:00",
  "Standort": {
    "GeraeteNr": "01",
    "PLZ": "48653",
    "Ort": "Coesfeld",
    "Strasse": "Boschstraße 14"
  },
  "Errors": [
    {
      "Key1": "value1"
    },
    {
      "Key2": "value2"
    }
  ]
}

I would like to display the Error Entrys as following: Errors

With the key difference being that the error entries should, in fact, be displayed on the left side. My first approach was to use multiline to condense the JSON file into a single line and then use JSON format to extract the labels. The problem I encountered is that the labels of the errors are independent of the key "Errors". So this approach doesn't seem to work. Is there another way for me to solve this problem? My approach: labels

0

There are 0 answers