any help is greatly appreciated.
I have fluentd running on kubernetes. I have logs coming from java application. The stack trace is multiline and the issue is each line of the multiline log is reported as a new log event.
below is my config. can someone advice what is wrong with the config? additionally, I am using containerd and not docker for running containers.
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Multiline On
Parser_Firstline multiline
Parser_1 cri
filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://k3s-master.test.com:6443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
K8S-Logging.Parser On
K8S-Logging.Exclude Off
[FILTER]
Name parser
Match kube.*
Key_Name log
Parser cri
Reserve_Data On
[FILTER]
Name parser
Match kube.*
Key_Name message
Parser json
Reserve_Data On
Preserve_Key On
output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
Logstash_Format On
Logstash_Prefix logstash-k3s-test
Replace_Dots On
Retry_Limit False
Type flb_type
parsers.conf: |
[PARSER]
Name json
Format json
[PARSER]
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep On
[PARSER]
Name multiline
Format regex
Regex ^(?<time>\d{4}[-\/]\d{2}[-\/]\d{2}.*) (?<stream>stdout|stderr) . (?<message>(\{|(ts=)?\[?\d{4}[-\/]\d{2}[-\/]\d{2}).*)$
sample log:
2020-10-15T08:48:37.932000736+02:00 stdout F 2020-10-15 08:48:37.923 ERROR [Service,a1793c4339d3f230,a1793c4339d3f230,true] 1 --- [nio-8080-exec-2] c.k.m.i.rest.GlobalExceptionHandler : Status Response Error: 304 NOT_MODIFIED
2020-10-15T08:48:37.932050633+02:00 stdout F
2020-10-15T08:48:37.932061926+02:00 stdout F org.springframework.web.server.ResponseStatusException: 304 NOT_MODIFIED
2020-10-15T08:48:37.932071245+02:00 stdout F at com.kobo.microservice.rest.InventoryController.queryInventory(InventoryController.java:70) ~[classes!/:1.197.1]
2020-10-15T08:48:37.932080389+02:00 stdout F at jdk.internal.reflect.GeneratedMethodAccessor280.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932088907+02:00 stdout F at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932096876+02:00 stdout F at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932113131+02:00 stdout F at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932121817+02:00 stdout F at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932131116+02:00 stdout F at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932140307+02:00 stdout F at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
According to fluentd documentation:
Original Events: