kubernetes pod logging broken with journald logging driver

Question

Looks like journald pod logging ('kubectl logs' command) doesn't work with kubernetes v1.18.x. Is there a way to make this work with v1.18.x? I've set up a multinode cluster with docker logging driver as 'journald' (using /etc/docker/daemon.json) and using systemd-journal-gatewayd with pull approach to aggregate historical logs. However, I'm very much interested in being able to tail current logs using 'kubectl logs' or, 'kubectl logs -l 'app=label' --prefix -f' (for cluster-wide logs).

# kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

# docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"

Running 'kubectl logs' gives the following error:

# kubectl logs <pod-name>
failed to try resolving symlinks in path "/var/log/pods/default_<pod-name>_xxxx/<container-name>/0.log": lstat /var/log/pods/default_<pod-name>_xxxx/<container-name>/0.log: no such file or directory

Answer 1

After you changed logging driver, the location of logs has changed to host's journal. So far the logs can be retrieved by docker but you have to let kubelet know about the changes.

You can do it by passing --log-dir=/var/log to kubelet. After adding the flag you have run systemctl daemon-reload and restart kubelet. It has to be done on all of the nodes.