Send logs to ELK container from Docker containers (without Filebeat)

Question

I'm using the SEBP/ELK Docker container as it appears to be the most suitable for quickly setting up application logging. Unfortunately, after reading through the docs there doesn't seem to be an easy way to get logs from sibling containers without the use of Filebeat.

I don't want to install Filebeat on each of my containers because that seems like it goes directly against Docker's separation of duties mantra.

TLDR; how do I get logs from my application containers to my ELK container?

Answer 1

SEBP/ELK was the wrong tool to tackle this problem. Instead, I should have been using a project that spins up a container for each of the elements of the ELK stack: Elasticsearch, Logstash, and Kibana. I found just such a repository on GitHub.

The deviantony/docker-elk project combines the three ELK elements into a working set of containers. The great thing about this is that unlike the SEBP/ELK project, deviantony/docker-elk doesn't take an opinionated view about what features should be available and what should be closed off. In the SEBP/ELK project, the ability to write to port 5000 is removed and when you try to add it back via a custom logstash.conf file, the UDP listener ultimately fails. Conversely, the deviantony/docker-elk project just works.

Bonus points: This project also has a branch that includes X-Pack which adds a minimal layer of security out of the box.

Answer 2

Using filebeat in each container is against Docker's philosophy. It will be waste of resources, And have more management overhead.

You can use local log file via logstash.

Example config:

input {
  file {
    path => "/var/log/apache.log"
    type => "apache-access"  # a type to identify those logs (will need this later)
    start_position => "beginning"
  }
}

Now we have to make the log files local to logstash container:

If you are using bind mounts you can mount the same directory in logstash container.

sudo docker run -d -v /path/to/logs/:/path/to/logs/in/container logstash

If you are using volumes you can mount the same volume that contains logs to logstash too.

sudo docker run -d -v logvol:/path/to/logs/in/container logstash