The Java module writes a single-line log:
{"timestamp":"2020-09-29 10:46:18.761","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}{"timestamp":"2020-09-29 10:49:11.338","level":"INFO","message":"status: OK","logger":"thrift.handler.CheckTokenThriftHandler","system":"-","service":"AUTH BE","conf_item":"-"}...
Each new event is pasted to the end of the line. In the fluentd settings I use the standard parser:
[PARSER]
Name java-system
Format json
Time_Key time
Time_Format %Y-%m-%d %H:%M:%S.%L
Time_Keep On
If I split the events in the log into lines, the agent sends everything to elastic without any problems. If everything is in one line, it doesn't send anything. It doesn't write anything informative in the log.
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.0 tag=gw_system
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.0:es.0
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.1 tag=calc-system
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.1:es.3
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.2 tag=push-system
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.2:es.2
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [trace] [router] input=tail.3 tag=gw_integration
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [router] match rule tail.3:es.1
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [ info] [sp] stream processor started
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log read=35484 lines=0
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log read=336548 lines=0
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log read=12302 lines=0
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.3] file=/opt/java_services/gateway/log/integration.log promote to TAIL_EVENT
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.0] file=/opt/java_services/gateway/log/system.log promote to TAIL_EVENT
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.1] file=/opt/java_services/calculator/log/system.log promote to TAIL_EVENT
Sep 29 15:11:41 td-agent-bit: [2020/09/29 15:11:41] [debug] [input:tail:tail.2] file=/opt/java_services/push/log/system_temp.log promote to TAIL_EVENT
Wrote a separate parser
[PARSER]
Name java_push_system
Format regex
Regex (?<date>(?<={"timestamp":")(\d+\-\d+\-\d+\s+\d+:\d+:\d+.\d+))",(?<message>[^}]*)
Time_Key time
Time_Format %Y-%m-%d %H:%M:%S.%L
Time_Keep On
On the site regex101.com my string is parsed, but the agent still does not send anything until it is split into separate lines.
/etc/td-agent-bit/conf]# cat push-system*
[FILTER]
Name record_modifier
Match push-system
Record hostname ${HOSTNAME}
Record environment DEV
Record path /opt/java_services/push/log/system.log
[INPUT]
Name tail
Tag push-system
Parser java-system
Path /opt/java_services/push/log/system.log
DB /etc/td-agent-bit/system-push.db
Buffer_Max_Size 32MB
Buffer_Chunk_Size 8MB
[OUTPUT]
Name es
Match push-system
Host 10.*
Port 9200
HTTP_User *
HTTP_Passwd *
Index push
Help to understand please!