Injest logs as JSON in Container Optimized OS

Question

I am able to injest logs to Google Log Viewer with the help of stackdriver logging agent from Container Optimized OS as JSON.

It injests logs as a value to message, but not as json payload with the default configuration

What I have tried?

I have changed the fluentd config in /etc/stackdriver/logging.config.d/fluentd-lakitu.conf to the following:

<source>
  @type tail
  format json
  path /var/lib/docker/containers/*/*.log
  <parse>
    @type json
  </parse>
  pos_file /var/log/google-fluentd/containers.log.pos
  tag reform_contain
  read_from_head true
</source>

But its unable to send logs to Log viewer

OS: Container Optimized OS cos-81-12871-1196-0

Answer 1

As @Kamelia Y mentioned about the https://issuetracker.google.com/issues/137517429

There is a mention on workaround used

<filter cos_containers.**>
@type parser
format json
key_name message
reserve_data false
emit_invalid_record_to_error false
</filter>

The above snippet parses the logs into JSON and injest to Cloud Logging.

In this discussion in Google Groups on Stackdriver, we have discussed on how to use it with startup-script.

Here is the snippet for startup script.

cp /etc/stackdriver/logging.config.d/fluentd-lakitu.conf /etc/stackdriver/logging.config.d/fluentd-lakitu.conf-save
# Shorter version of the above: cp /etc/stackdriver/logging.config.d/fluentd-lakitu.conf{,-save}
(
head -n -2 /etc/stackdriver/logging.config.d/fluentd-lakitu.conf-save; cat <<EOF
<filter cos_containers.**>
@type parser
format json
key_name message
reserve_data false
emit_invalid_record_to_error false
</filter>
EOF
) > /etc/stackdriver/logging.config.d/fluentd-lakitu.conf
sudo systemctl start stackdriver-logging

This image can be used to generate random JSON logs. https://hub.docker.com/repository/docker/patelathreya/json-random-logger

Answer 2

I've found this issue on Google's Public Issue Tracker which discusses the same problem you mentioned in your use case. Google Product team has been notified about this limitation and they are working on it. You just have to go there and click on the star next to the title so you get updates on the issue and you give the issue more visibility.