Summary: I want the logging activity in my docker containers to be sent to the ELK stack. What I receive is a message intended to be JSON, split into individual lines.

I configured logging on docker:

# cat /etc/docker/daemon.json
{
    "log-driver": "gelf",
    "log-opts": {
        "gelf-address": "udp://logstash:12201"
    }
}

Logstash is configured to receive GELF messages and then forward them to Elasticsearch (it itself runs in a docker container):

# cat /etc/docker/logstash/gelf.yml
input {
  gelf {
    type => docker
    port => 12201
  }
}
output {
  stdout {}
  elasticsearch {
    hosts => ["http://elasticsearch:80"]
  }
}

I receive messages in Elasticsearch, but they are all split line by line, instead of being sent as a JSON block:

enter image description here

The details of one of these lines (the 3rd one - "container_name" => "logstash"):

{
  "_index": "logstash-2019.05.28-000001",
  "_type": "_doc",
  "_id": "-I4b_moBBedB4gDKLglO",
  "_score": 1,
  "_source": {
    "source_host": "192.168.10.2",
    "host": "srv",
    "created": "2019-05-28T11:01:16.452463192Z",
    "tag": "2a6e38842912",
    "container_id": "2a6e38842912d8d7033a5db5d4fa8ac0c19df3a8bfe667607323fd12df5e705f",
    "@timestamp": "2019-05-28T11:02:34.212Z",
    "@version": "1",
    "message": "    \"container_name\" => \"logstash\",",
    "image_id": "sha256:93ae8cd115605387515d96373bd16709c507376fe0c73dd125505763ffdf0500",
    "level": 6,
    "command": "/usr/local/bin/docker-entrypoint",
    "version": "1.1",
    "image_name": "docker.elastic.co/logstash/logstash:7.1.0",
    "container_name": "logstash",
    "type": "docker"
  },
  "fields": {
    "created": [
      "2019-05-28T11:01:16.452Z"
    ],
    "@timestamp": [
      "2019-05-28T11:02:34.212Z"
    ]
  }
}

What is wrong with the logstash setup?

0 Answers