log4j2 configuration for graylog

Question

We want to centralize all our java application logs on Graylog server. We use apache tomcat as a container and log4j for the logging framework. log4j2.xml

        <Socket name="GELF" protocol="tcp" host="graylog.domain.com" port="12201">
            <GelfLayout host="%host">
                    <KeyValuePair key="projectName" value="magus"/>
                    <KeyValuePair key="level" value="%level"/>
                    <KeyValuePair key="timestamp" value="%d"/>
                    <KeyValuePair key="server" value="%host"/>
                    <KeyValuePair key="logStream" value="magus"/>
                    <KeyValuePair key="version" value="1.1"/>
                    <KeyValuePair key="projectName" value="magus"/>
                    <KeyValuePair key="logStream" value="magus"/>
                    <KeyValuePair key="className" value="%C"/>
                    <KeyValuePair key="simpleClassName" value="%C{1}"/>
            </GelfLayout>
    </Socket>

Loggers

    <Loggers>
       <Root level="INFO">
          <AppenderRef ref="CONSOLE"/>
          <AppenderRef ref="LOG"/>
          <AppenderRef ref='GELF'/>
       </Root>
    </Loggers>

log detail

2021-01-26 20:05:01,343 http-nio-31381-exec-1 DEBUG Reconnecting /graylog.domain.com:12201
2021-01-26 20:05:01,344 http-nio-31381-exec-1 DEBUG Creating socket /graylog.domain.com:12201
2021-01-26 20:05:01,344 http-nio-31381-exec-1 DEBUG Closing SocketOutputStream    java.net.SocketOutputStream@8cb01fa
2021-01-26 20:05:01,345 http-nio-31381-exec-1 DEBUG Connection to graylog.domain.com:12201 reestablished: Socket[addr=/graylog.domain.com,port=12201,localport=41482]

As you see my application create a socket connection wiith gray log server. But we did not see any log on the Gray log server

versions
tomcat - 9.0.16.0
jdk - 1.8.0_201-b09(64 bit)
log4j2 - 1.12 / 1.14(both checked)
os - Linux 3.10.0-1062.12.1.el7.x86_64
gray log - Graylog 3.0.2+1686930 on graylogsrv (Oracle Corporation 1.8.0_232 on Linux 3.10.0-1062.9.1.el7.x86_64)

documentation https://logging.apache.org/log4j/2.x/manual/layouts.html#GELFLayout

I want to use log4j2 other than extrnal library like logstash-gelf

UPDATE
this is the gray log server log

2021-01-27T12:18:45.079+04:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=45f04b90-6078-11eb-80bf-00505696a882, journalOffset=2771397770, codec=gelf, payloadSize=11, timestamp=2021-01-27T08:18:45.065Z, remoteAddress=/graylog.domain:58258} on input <600ecd97f7c4b60478f4504e>.
2021-01-27T12:18:45.079+04:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=45f04b90-6078-11eb-80bf-00505696a882, journalOffset=2771397770, codec=gelf, payloadSize=11, timestamp=2021-01-27T08:18:45.065Z, remoteAddress=/graylog.domain:58258}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('�' (code 65533 / 0xfffd)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: �Y�n�8��h; line: 1, column: 2]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702) ~[graylog.jar:?]
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558) ~[graylog.jar:?]
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456) ~[graylog.jar:?]
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1906) ~[graylog.jar:?]
    at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:749) ~[graylog.jar:?]
    at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3850) ~[graylog.jar:?]
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3799) ~[graylog.jar:?]
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2397) ~[graylog.jar:?]
    at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:127) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
    at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

how can we get original data to find the error?

Answer 1

Finally solved. According to documentation

GELF TCP does not support compression due to the use of the null byte (\0) as frame delimiter.

So after disabling compress on the log4j2 configuration we saw our log on the gray log server. The below code snippet is a working example

<Socket name="GELF" protocol="tcp" host="graylog.domain.com" port="12201">
            <!-- gelf tcp does not support compression-->
            <GelfLayout includeStackTrace="true" host="${hostName}" includeThreadContext="true" includeNullDelimiter="true"
                  compressionType="OFF">
                
                <KeyValuePair key="host" value="${hostName}"/>
                <KeyValuePair key="version" value="1.1"/>
                <!--<KeyValuePair key="short_message" value="$${event:Message}"/>--><!--   not required             -->
                <KeyValuePair key="application_name" value="${web:contextPathName}"/>
                <KeyValuePair key="thread_id" value="$${event:ThreadId}"/>
                <KeyValuePair key="thread_name" value="$${event:ThreadName}"/>
                <KeyValuePair key="timestamp" value="$${event:Timestamp}"/>
                <!--<KeyValuePair key="level" value="1"/>--><!-- default level type is number so we don't use-->
                <KeyValuePair key="log_level" value="$${event:Level}"/><!-- for readabilty and filtering-->
            </GelfLayout>
        </Socket>