Send GELF messages via HTTP to Logstash

Question

I am trying to send GELF messages (via HTTP client) to Logstash.

The logstash.conf file looks like this:

input {
  gelf {
    use_tcp => true
  }
}

output {
  elasticsearch { hosts => ["10.32.0.80:9200"] }
}

In Grafana, the log is displayed like this:

As you can see, I am getting a _jsonparsefailure tag which is also logged withing the logstash logs.

The logstash log:

[INFO ] 2021-04-14 14:36:54.155 [Agent thread] agent - Pipelines running {:count=>1, 
:running_pipelines=>[:main], :non_running_pipelines=>[]}
[ERROR] 2021-04-14 14:51:12.025 [Ruby-0-Thread-39: :1] gelf - JSON parse failure. Falling back to 
plain-text {:error=>#<LogStash::Json::ParserError: Unrecognized token 'PUT': was expecting ('true', 
'false' or 'null')
at [Source: (byte[])"PUT / HTTP/1.1
Content-Type: application/json; charset=utf-8
Host: 10.32.8.133:12201
Content-Length: 371
Expect: 100-continue
Connection: Keep-Alive

{"version":"1.1","host":"postch-msm-order-statistics-test","short_message":"B2, total shipments: 
999","full_message":"National: 999\nInternational: 0\nCollect: 
0","level":6,"_event_type":"ShipmentDeclarationSent","_order_hash":"RGhktM2k6iH7n5MbxzzAsQ==","_product_type":"B2","_national_shipments_count":999,"_international_shipments_count":0,""[truncated 28 bytes]; line: 1, column: 5]>, :data=>"\"PUT / HTTP/1.1\\r\\nContent-Type: application/json; charset=utf-8\\r\\nHost: 10.32.8.133:12201\\r\\nContent-Length: 371\\r\\nExpect: 100-continue\\r\\nConnection: Keep-Alive\\r\\n\\r\\n{\\\"version\\\":\\\"1.1\\\",\\\"host\\\":\\\"postch-msm-order-statistics-test\\\",\\\"short_message\\\":\\\"B2, total shipments: 999\\\",\\\"full_message\\\":\\\"National: 999\\\\nInternational: 0\\\\nCollect: 0\\\",\\\"level\\\":6,\\\"_event_type\\\":\\\"ShipmentDeclarationSent\\\",\\\"_order_hash\\\":\\\"RGhktM2k6iH7n5MbxzzAsQ==\\\",\\\"_product_type\\\":\\\"B2\\\",\\\"_national_shipments_count\\\":999,\\\"_international_shipments_count\\\":0,\\\"_collect_shipments_count\\\":0}\""}

Does anyone know why I am getting this error? I believe the Logstash Gelf Input plugin must know how the HTTP request (with the gelf message as body) looks like. Why am I getting this parsing error though? I have to solve this without using any filter. Any ideas?

Answer 1

Graylog supports multiple input types. It will accept GELF over UDP, TCP, or HTTP. The logstash gelf input does not support HTTP requests, just NUL delimited messages over UDP or TCP. You can use an HTTP input instead. The JSON will automatically be parsed using the json codec if the content type is application/json.

The input remaps a couple of fields, and optionally strips underscores from field names. You could use a ruby filter to implement the same changes in your pipeline.