I want to stream my app log messages to kafka using log4j kafka appender.

My application is using spring boot and log4j2.

I am seeing startup log messages in the topic but not the log statements used in the application.

Did i miss anything in the log4j2 configuration?

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="debug" name="kafka-logs" 
packages="myapp.kafka.test">
<Appenders>
    <Kafka name="Kafka" topic="test-kafka-topic">
        <JSONLayout />
        <Property name="bootstrap.servers">server1,server2</Property>
    </Kafka>
    <Async name="Async">
        <AppenderRef ref="Kafka"/>
    </Async>

    <Console name="stdout" target="SYSTEM_OUT">
        <PatternLayout pattern="%d{HH:mm:ss.SSS} %-5p [%-7t] %F:%L - %m%n"/>
    </Console>

</Appenders>
<Loggers>
    <Root level="INFO">
        <AppenderRef ref="Kafka"/>
        <AppenderRef ref="stdout"/>
    </Root>
    <Logger name="org.apache.kafka" level="WARN" />
</Loggers>

pom.xml

    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>2.11.2</version>
    </dependency>

    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.11.2</version>
    </dependency>

    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-simple</artifactId>
        <version>1.7.10</version>
    </dependency>

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-log4j-appender</artifactId>
        <version>1.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Java Code

@SpringBootApplication
public class TestApplication {
private static final Logger logger = 
LoggerFactory.getLogger(TestApplication.class);
public static void main(String[] args) {
  SpringApplication application = new SpringApplication(TestApplication.class);
  logger.info("Before Application Startup");
  application.run(args);
  logger.info("After Application Startup");
}
}

Logs in Kafka:

  {
  "thread" : "main",
  "level" : "INFO",
  "loggerName" : "TestApplication",
  "message" : "Started TestApplication in 17.19 seconds (JVM running for 22.67)",
  "endOfBatch" : false,
  "loggerFqcn" : "org.apache.commons.logging.LogFactory$Log4jLog",
  "instant" : {
  "epochSecond" : 1555422668,
  "nanoOfSecond" : 856000000
},
"threadId" : 1,
"threadPriority" : 5
}

Console output:

08:51:08.856 [main] INFO  TestApplication - Started TestApplication in 17.19 seconds (JVM running for 22.67)
[main] INFO TestApplication - Testing my application from TestApplication
[main] ERROR TestApplication - Testing my application from TestApplication

1 Answers

0
sendon1982 On

I think log4j2 already have the Kafka appender. As for me, I added Kafka client in gradle. So no need to add kafka-log4j-appender

compile group: 'org.apache.kafka', name: 'kafka-clients', version: '2.0.1'

Maven should be similar