Storing Laravel Logs using InfluxDb, Telegraf and Grok

Question

I'm trying to centralize all Laravel logs using Telegraf and visualize them with Grafana. For now I can store logs with the following format:

[2022-03-31 12:08:50] dev.INFO: microservice_name | default | Update User | 32 | [email protected] | Center Name | {"idMember":21212,"card":"00000000D","name":"PAUL","surname_1":"SURNAME1","email":"[email protected]"}

I'm using this grok pattern to parse it:

  name_override = "influxDb_users_log"
  grok_patterns = ["%{CUSTOM_LOG}"]
  grok_custom_patterns = '''
    MAGICDATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}
    CUSTOM_LOG \[%{MAGICDATE:date}\] %{DATA:env}\.%{DATA:severity}: %{DATA:microservice} \| %{DATA:client} \| %{DATA:message} \| %{DATA:id_member} \| %{DATA:email} \| %{DATA:club} \| %{GREEDYDATA:user_row}

Then I make this Flux Query

from(bucket: "bucket")
  |> range(start: 2018-05-22T23:30:00Z, stop: 2023-05-23T00:00:00Z)
  |> filter(fn: (r) => r["_measurement"] == "influxDb_users_log")
  |> filter(fn: (r) => r["_field"] == "client" or r["_field"] == "club" or r["_field"] == "env" or r["_field"] == "email" or r["_field"] == "date" or r["_field"] == "microservice" or r["_field"] == "user_row" or r["_field"] == "severity" or r["_field"] == "message" or r["_field"] == "id_member")

Some results appear but I will not obtain a table similar to what I would expect doing a SQL Query where each line of the log would be parsed in its column and shown in one line. instead I'm obtaining this.

When later I want to obtain queries using the PHP library I obtain results in a strange format like this:

$logs = $queryApi->queryRaw(
    'from(bucket:"bucket")
        |> range(start: -100h)');

dd($logs);

Am I doing something wrong? Am I storing incorrectly the values? Thanks!

Answer 1

To get a table with all fields (client, message etc) in one row, use schema.fieldsAsCols(). InfluxDB 2.x stores fields in separate tables. This function is a convenience function for pivot() used to organize values column-wise, like a response from SQL query.

queryRaw returns unprocessed response ie. Flux annotated CSV, so you probably want to use query to get a table of records. Please see Queries documentation.